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Auditor General of Canada 
Vérificatrice générale du Canada 


To the Honourable Speaker of the House of Commons: 


| have the honour to transmit herewith this 2009 annual report to the House of Commons, which is 
to be laid before the House in accordance with the provisions of subsection 7(3) of the Auditor General Act. 


SA roreere eter 


Sheila Fraser, FCA 
Auditor General of Canada 


OTTAWA, 3 November 2009 
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Matters of Special 
Importance—2009 


Matters of Special Importance—2009 


I am pleased to present my 2009 Fall Report to the House of 
Commons. The Commissioner of the Environment and Sustainable 
Development is also presenting a report. 


This Report addresses a wide range of issues typical of the challenges 
facing government today: 


* coordinating appropriate responses to emergencies, 
such as pandemics and terrorist attacks; 


* selecting foreign workers at a time when industrialized countries 
are competing to attract qualified labour; 


¢ providing effective international aid; 


© Photo: Couvrette/Ottawa 


Sheila Fraser, FCA a 


as ie . ; 
ck are pene ae supporting the transition to electronic health records; 
* increasing First Nations’ control over land management as 


a critical component of their future well-being; 


* buying military vehicles to protect our troops in Afghanistan 
from improvised explosive devices; 


* ensuring that income tax legislation is clear and up to date; and 


¢ knowing whether government programs are achieving their 
intended results or need to be adjusted. 


The Main Points from our chapters on all of these issues are included 
at the end of these comments. 


While the issues are diverse, a common message emerges in several 
of our chapters: how important it is, when the government designs 
programs, develops policies, and makes commitments, that it fully 
analyze the practical challenges of implementing them successfully. 
Our findings show that this is not always the case. 


Despite good intentions, there are examples of policies adopted, 
programs launched or changed, and commitments made without a 
full analysis of the risks involved, the resources needed, the potential 
impact on other players, and the steps required to achieve the desired 
results. We also see examples where there is no long-term vision or 
strategy to guide a department’s overall programming, and others 
where there is no ongoing evaluation of program effectiveness. The 
result can be a fragmented approach to programming in response to a 
problem of the day, creating other problems that were not anticipated. 


_ 
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A prime example is immigration programming, described in Chapter 2, 
Selecting Foreign Workers Under the Immigration Program. 
Canada has an ongoing need for permanent and temporary workers 
with various skills, and it must compete with other countries to attract 
them. It is critical that the government’s programs to facilitate the 
entry of these workers into Canada be designed to meet the needs of 
the Canadian labour market. 


We found that Citizenship and Immigration Canada (CIC) has made.a 
number of key decisions without first properly assessing their costs and 
benefits, potential risks, and potential impacts on other programs. and 
delivery mechanisms. For example, program changes in recent years 
have resulted in a significant shift in the types of workers being 
admitted permanently to Canada. We saw little evidence that this 
shift is part of any clear strategy to best meet Canada’s labour needs. 
A strategic roadmap for the future, such as the national framework the 
Department committed to develop in 2004, would help to provide a 
clear vision of what each program is expected to contribute to the 
economic objectives for immigration. 


In addition, the Department did not carry out sufficient analysis to 
support its strategy for reducing the inventory of applications to enter 
Canada under the federal skilled worker category. While it is too early 
to assess their full impact, there are indications that the measures 
taken might not have the desired effect. 


This chapter also provides an example of how not fully considering the 
way a policy will be carried out can lead to problems in implementing it. 
The Immigration and Refugee Protection Regulations state that before 
issuing a work permit to a temporary foreign worker, a CIC officer must 
assess, on the basis of an opinion provided by Human Resources and 
Skills Development Canada (HRSDC), if the job offer is genuine and 
not likely to negatively affect the labour market in Canada. 

The Regulations state the factors to consider in assessing labour market 
effects but are silent on how to assess whether a job offer is genuine. 
We found that CIC and HRSDC—who co-manage temporary foreign 
worker programs—were not clear about their respective roles in making 
this assessment and how it is to be done. As a result, work permits could 
be issued for jobs or employers that do not exist. 


Chapter 7, Emergency Management, also illustrates the practical 
challenges of implementing policies. Public Safety Canada is the 
coordinating agency for federal emergency management activities. 
The aim is to provide a federal focal point to eliminate potential 
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confusion when responding to crises such as the HIN1 pandemic, 
widespread forest fires, and the 1998 ice storm in Eastern Canada. 


The expertise and experience to deal with various aspects of emergencies 
reside in several departments, and it is up to each department to 
determine in accordance with its mandate how it will address an 
emergency. Under the Emergency Management Act, however, Public 
Safety Canada has the responsibility to coordinate the emergency 
management activities of federal government institutions along with 
those of the provinces and territories. 


We found that it is not yet clear how Public Safety Canada will carry 
out these responsibilities, as the Federal Emergency Response Plan has 
not been adopted and endorsed by the government and the federal 
departments involved. Therefore, it is not clear how, in practice, Public 
Safety Canada will reconcile its role as the focal point for federal action 
in a crisis with its lack of authority to dictate what other departments 
will do, given the principle that departments are accountable to their 
own ministers. | 


The Department also has the responsibility to promote a common 
approach to emergency management, including the adoption of 
standards and best practices. We noted that emergency first 
responders—police officers, firefighters, and medical and ambulance 
crews—are limited in their ability to communicate and work together 
in an emergency because their voice communication systems, such as 
radios, are not compatible. Public Safety Canada officials told us that 
their role is not to develop standards but rather to assist groups in 
developing their own—and very little progress has been made to date. 
It is not clear how helping groups develop their own standards would 
promote a common approach. 


In addition, we were told by officials that while the Department can 
provide advice and coordination, it is up to operational departments to 
identify Canada’s critical infrastructure and determine how it should 
be protected before a coordinated approach can be implemented. 
However, Public Safety Canada has provided no advice to departments 
to ensure that they do this. 


Public Safety Canada has made some progress, slowly, in the areas of 
strategy, framework, and plan development. However, until it clarifies 
what its responsibility for coordination means in practical terms, it is 
unclear to us—and to some of the departments that have asked it for 
what concrete steps it would take to coordinate federal 


guidance 
action in the event of an actual emergency of national significance. 


Report of the Auditor General of Canada—Fall 2009 


T, 3 


MATTERS OF SPECIAL IMPORTANCE—2009 


In Chapter 8, Strengthening Aid Effectiveness, we note that in the 
absence of a comprehensive strategy to guide the aid efforts of the 
Canadian International Development Agency (CIDA), plans and 
priorities often change before they are fully implemented. 


CIDA’s 2002 Policy Statement on Strengthening Aid Effectiveness 
commits it to principles of aid effectiveness adopted by the international 
donor community: align efforts with recipients’ needs and priorities; 
harmonize activities with those of other donors; and use new forms of aid 
known as program-based approaches, in which donors coordinate 
support to the budgets of recipient governments or local organizations 
for a development program delivered using local systems and procedures. 
The international donor community has-agreed that applying these 
principles improves the effectiveness of aid. 


In the countries we examined, we found that CIDA is working with 
other donors to apply the principles of aid effectiveness. We were told 
by donors and by officials of recipient governments that CIDA staff in 
the field are highly regarded and their efforts are appreciated. 


In examining corporate management processes, however, we found 
that while some action was taken to align programming with the 
principles of aid effectiveness, the actions were selective and not 
guided by a comprehensive strategy. In some cases, initiatives were 
begun but not tracked to completion. In other cases no action plans 
were developed at all. There is little evidence that senior managers 
systematically reviewed the implementation of the 2002 Policy 
Statement. Shifting priorities and a lack of clear direction and action 
plans led to a situation in which donors, recipient governments, and 
CIDA program staff are unclear about the Agency’s direction 

and long-term commitment to specific countries or regions. 


The international donor community has recognized that the long-term 
nature of international development requires stability and 
predictability of programming. In our view, frequent changes in policy 
direction and substantial turnover of senior personnel in recent years 
have posed significant challenges for CIDA’s aid effectiveness agenda. ° 


The challenges of supporting the implementation of electronic health 
records nationwide are discussed in Chapter 4, Electronic Health 
Records. The audit looked at Canada Health Infoway (Infoway), 

a not-for-profit corporation created to lead the national development 
of electronic health records (EHRs) with funding from Health Canada. 
We found that Infoway has accomplished much in the eight years since 
it was created. . 
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Developing and implementing EHRs that are compatible across 
Canada requires national leadership and the collaboration of many 
stakeholders, including health care professionals. Chapter 4 notes that 
while Infoway has met a number of important challenges, others 
remain. While our discussion of future challenges is not based directly 
on our audit work, we have raised them to indicate the magnitude of 
the efforts still needed. 


For example, Infoway has set a goal of having EHRs for 50 percent of 
Canadians available to their authorized health care professionals 

by 2010. However, by 31 March 2009, only 17 percent of Canadians 
were living in a province or territory where a complete EHR system 
was available. 


Further, not all completed EHR projects have implemented the 
standards required for national compatibility. According to Infoway, 
some of them will do so only when they are upgraded in the future. 
It is not clear who will provide the funding for this, and when. 


Infoway, Health Canada, and the provinces and territories need to 
carefully consider how these and other challenges can be met so the 
EHR initiative can deliver the intended results. 


The process of developing new programs or making changes to existing 
ones can benefit significantly by using information from effectiveness 
evaluations, as discussed in Chapter 1, Evaluating the Effectiveness 
of Programs. 


Done well, effectiveness evaluations provide reliable, objective 
information that helps identify programs that are working as intended, 
those that are no longer needed, and those that are not achieving their 
objectives and could be replaced by programs that will achieve them 
more cost-effectively. 


Unfortunately, the history of program evaluation in the federal 
government shows a poor record of implementation. After 

four decades of effort, evaluations are still not providing enough 
reliable evidence about whether program objectives are being met. 
In the absence of such information, it is difficult for departments to 
identify problem areas and correct them in order to make their 


programs more effective. 


Our chapter on selecting foreign workers notes that evaluations of the 
programs we audited are either outdated or have not been done at all. 
Consequently, not only are CIC and HRSDC unable to determine to 
what extent the programs are meeting their objectives and achieving 
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the expected outcomes, they are missing valuable information that 
could have been used in the recent past and could be used in the 
future when changing programs or designing new ones. 


The departments we examined have concerns about their capacity to 
implement the requirement in the 2009 Policy on Evaluation that all 
direct program spending be evaluated. An earlier requirement to 
evaluate all grant and contribution programs had already taxed their 
capacity. In our view, identifying programs where effectiveness 
information can be put to the best use will be a key part of 
implementing the coverage requirements of the policy. 


Conclusion 


Our findings this year underscore the importance of thinking through 
the implementation challenges when policies and programs are 
developed or changed. Having a complete picture of what needs to 
be done, by whom and when, how much it will cost, what risks are 
involved, how other programs might be affected, and what has or has 
not worked well to date, can make the difference between a program 
that meets its objectives and one that does not, one that delivers 
results for Canadians and one that does not. Several of our 
recommendations to departments address these issues. 


I hope members of Parliament will find this report useful in holding 

the government to account for its management of public funds and its 
delivery of services to Canadians. I look forward to continuing to work 
with parliamentary committees as they review the results of our audits. 


As always, I wish to thank my colleagues and staff for their effort and 
their ability to deal with the many challenges we face together. 
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Chapter 1 


What we examined 


Why it’s important 
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Evaluating the Effectiveness 
of Programs 


Main Points 


Effectiveness evaluation is an established tool that uses systematic 
research methods drawn from many different disciplines to assess how 
well a program is achieving its objectives. When done well, it allows 
departments to develop evidence to determine how well their 
programs meet expectations, and whether they are cost-effective. Over 
the past 40 years, the federal government has made efforts to embed 
the management practice of evaluating program effectiveness, as an 
essential part of its support for program evaluation. 


The 2006 Federal Accountability Act enacted into law a requirement 
that all grant and contribution programs be evaluated every five years. 
The new Policy on Evaluation that came into effect in 2009 extends 
the requirement for evaluation to cover all direct program spending 
over a five-year cycle. 


We examined how evaluation units in six departments identify and 
respond to the various needs for effectiveness evaluations. We also 
looked at whether they have built the required capacity to respond to 
those needs. In addition, we looked at the oversight and support role of 
the Treasury Board of Canada Secretariat in monitoring and improving 
the evaluation function in the government, specifically with respect to 
effectiveness evaluations. The period covered by our audit was 2004 


to 2009. 


Governments are under continual pressure to spend money on a range 
of programs designed to serve particular needs of society. While many 
factors affect the decisions that governments must ultimately make 


about programs, effectiveness evaluations can aid their decision 


making by providing objective and reliable information that helps 
identify programs that are working as intended; those that are no 
longer needed; and those that are not accomplishing the desired 
objectives and could be replaced by programs that will achieve the 
objectives more cost-effectively. In addition, effectiveness evaluation is 


expected to serve the information needs of parliamentarians. 


Chapter 1 i 9 


Main Points—CHuaPTERS 1 To 8 


What we found 


One of the most important benefits of effectiveness evaluation is to 
help departments and agencies improve the extent to which their 
programs achieve their objectives. Departments need to demonstrate 
to Parliament and taxpayers that they are delivering results for 
Canadians with the money entrusted to them. 


- The six-departments we examined followed systematic processes to 
plan their effectiveness evaluations and completed most of the 
evaluations they had planned. However, over the audited period, 
each department’s evaluations covered a relatively low proportion of 
its total program expenses—between five and thirteen percent 
annually across the six departments. 


- In effect, the rate of coverage was even lower because many of the 
effectiveness evaluations we reviewed did not adequately assess 
program effectiveness. Often departments have not gathered the 
performance information needed to evaluate whether programs are 
effective. Of the 23 evaluation reports we reviewed, 17 noted that 
the analysis was hampered by inadequate data, limiting the 
assessment of program effectiveness. 


¢ The departments we examined told us that it remains a challenge to 
find experienced evaluators, and they have made extensive use of 
contractors to meet requirements. Departments expressed concern 
about their capacity to start in 2013 to evaluate all direct program 
spending, as required by the 2009 Policy on Evaluation. To ensure 
full coverage (which includes grants and contributions), they will 
have to evaluate an average of 20 percent of their direct program 
spending each year of the five-year cycle. 


- The Treasury Board of Canada Secretariat introduced initiatives 
over the past few years to address the need for improvements in 
evaluation across the government. However, it did not provide 
sustained support for effectiveness evaluation. In particular, it made 
little progress on developing tools to assist departments with the 
long-standing problem of a lack of sufficient data for evaluating 
program effectiveness. 


* With the exception of Environment Canada, which has processes in 
place to identify needed improvements, the audited departments do 


not regularly identify and address weaknesses in effectiveness 
evaluation, 


The Secretariat and the departments have responded. The Treasury 
Board of Canada Secretariat and the departments agree with our 
recommendations. Their detailed responses follow each 
recommendation throughout the chapter. 
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What we examined 


Why it’s important 


Selecting Foreign Workers Under 
the Immigration Program 


Main Points 


In Canada, the federal government and the provinces and territories 
share jurisdiction over immigration. Citizenship and Immigration 
Canada (CIC) is generally responsible for the selection of immigrants 
and other foreign nationals and for ensuring that they are admissible— 
that is, that they do not present any risk to the health and safety of 
Canadians. The Department has also signed agreements with most 
provinces and territories allowing them to play an active role in 
selecting immigrants to meet the specific needs of their labour markets. 


In 2008, Canada admitted about 250,000 people as permanent 
residents, including about 150,000 individuals and their immediate 
family members selected on the basis of attributes that would enable 
them to succeed in a dynamic labour market, such as education, 
professional experience, and official language ability. In addition, 
Canada allowed almost 370,000 temporary foreign workers in 2008 to 
fill a short-term need for labour. 


We examined how CIC plans for and manages programs designed to 
facilitate the entry of permanent and temporary workers into Canada 
and the recognition of foreign credentials in Canada. In addition, we 
looked at the role of Human Resources and Skills Development 
Canada (HRSDC) in supporting the planning and delivery of these 
programs, including the issuance of labour market opinions by its 
Service Canada offices. The audit covered the period from June 2002, 
when the Immigration and Refugee Protection Act came into effect, to 


30 June 2009. 


Our audit did not cover how CIC assesses whether applicants are 
admissible to Canada or how the provinces and territories nominate 
candidates for selection. Nor did we examine the Canada Border 
Services Agency’s processing of work permit applications at points of 


entry into Canada. 


Immigration has played an important role in the economic, social, and 
cultural development of Canada throughout our history. Its role is just 


as important today, given our aging population and labour force. 
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What we found 


Canada has an ongoing need for permanent workers with various skills 


and must compete with other countries to attract them. In addition, 


Canada has a need for various types of temporary workers to address 


short-term needs of the labour market, which vary from year to year 


and from region to region of the country. 


It is critical that the government’s programs to facilitate the entry of 


permanent and temporary workers be designed and delivered in a way 


to ensure that the right people are available at the right time to meet 
the needs of the Canadian labour market. The choices that are made 
now will affect the kind of society Canada has in the future. 


Although CIC followed a sound decision-making process in 2008 to 
design the Canadian Experience Class (a category of skilled foreign 
workers and students with Canadian work experience), the 
Department has made other key decisions without properly assessing 
their costs and benefits, risks, and potential impacts on other 
programs and delivery mechanisms. Program changes in recent years 
have resulted in a significant shift in the types of workers being 
admitted permanently to Canada under the immigration program’s 
economic component. We saw little evidence that this shift is part of 
any well-defined strategy to best meet the needs of the Canadian 
labour market. 


The inventory of applications in the Federal Skilled Worker category 
has almost doubled since our 2000 audit and, in December 2008, 
represented more than 620,000 people waiting an average of 

63 months for a decision on whether they would be admitted. 
Measures taken by CIC in 2008 to limit the number of new 
applications—for example, processing only those that meet new, 
more narrowly defined criteria—were not based on sufficient analysis 
of their potential effects. While it is too early to assess their full 


- impact, trends in the number of applications received since the 


beginning of 2009 indicate that they might not have the desired 
effect, and CIC could be unable to process new applications within 
the 6 to 12 months it has forecast. Furthermore, CIC does not know 
and has not defined how much time it should take to clear the 
inventory of applications on hand when the measures were 
introduced. 


CIC and HRSDC have not clearly defined their respective roles and 
responsibilities in assessing the genuineness of job offers and how + 
that assessment is to be carried out. As a result, work permits could 
be issued to temporary foreign workers for employers or jobs that do 
not exist. In addition, there is no systematic follow-up by either 
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department to verify that in their previous and current employment 
of temporary foreign workers, employers have complied with the 
terms and conditions (such as wages and accommodations) under 
which the work permits were issued. This creates risks to program 
integrity and could leave many foreign workers in a vulnerable 
position, particularly those who are physically or linguistically 
isolated from the general community or are unaware of their rights. 
Furthermore, weaknesses in the practices for issuing labour market 
opinions raise questions about the quality and consistency of 


decisions being made by HRSDC officers. 


¢ CIC has successfully introduced a number of initiatives and tools to 
address some of the inefficiencies we reported in 2000 in its 
processing of applications in missions overseas. However, efficiency 
gains will be seriously limited until an information technology system 
that has been under development for almost 10 years is implemented 
in missions abroad and CIC makes effective use of available 
technologies. In the meantime, employees in missions abroad are still 
buried in paperwork and spending a great deal of their time on 
clerical tasks. In addition, while the Department has developed a 
quality assurance framework that is available to all missions, 
immigration program managers are not required to use it or to report 
on quality assurance. Therefore, CIC still has little assurance that 
overall, decisions by visa officers are fair and consistent. 


The entities have responded. The entities agree with all of the 
recommendations. Their responses follow each recommendation 
throughout the chapter. 
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Why it’s important 


Income Tax Legislation 


Main Points 


The Department of Finance Canada is responsible for formulating tax 
policy at the federal level and for developing the legislation and 
regulations needed to implement federal tax measures. In the past, the 
Department has developed legislative amendments to address 
technical deficiencies in the Income Tax Act, in order to more clearly 
convey the intent of the legislation or to make it easier for taxpayers to 
comply with it. Packages of technical amendments have first been 
released for public comment and then tabled in Parliament in the form 
of a technical bill. Usually these amendments do not involve any 
reconsideration of tax policy. 


The Canada Revenue Agency is responsible for applying and 
interpreting the Income Tax Act—one of the longest and most complex 
pieces of Canadian legislation. The primary goal of the Agency as 
Canada’s tax administrator is to ensure that taxpayers comply with 
their obligations and that Canada’s tax base is protected. 


We looked at how the Department of Finance Canada develops 
technical amendments for tabling in Parliament; we did not examine 
the process for developing legislative amendments that involve 
changes in tax policy. 


In addition, we examined how the Canada Revenue Agency provides 
taxpayers and its tax auditors with guidance on the application and 
interpretation of the Income Tax Act, and how it assists the Department 
of Finance Canada in-identifying and developing technical changes 
that may be needed in the legislation. The audit covered the period 
from 2004 to 2009, with relevant historical information reviewed back 


to 1973. 


Canada’s tax system relies on taxpayers to self-assess and pay the 
income taxes they owe. According to the Canada Revenue Agency, 
most taxpayers will meet their tax obligations if given the proper tools 
and information. 


Taxpayers’ ability to comply with tax legislation depends on their 
understanding of how the rules apply to their own circumstances. 
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When the intent of the legislation is not clearly conveyed by the 
words, taxpayers may find it difficult to assess the income taxes they 
owe and this could foster tax avoidance. Uncertainty about how the 
law should be applied can also add to the time taken and costs incurred 
by tax audits and tax administration. 


What we found = + No income tax technical bill has been passed since 2001. Although 
the government has said that an annual technical bill of routine 
housekeeping amendments to the Act is desirable, this has not 
happened. As a result, the Department of Finance Canada has a 
backlog of at least 400 technical amendments that have not been 
enacted, including 250 “comfort letters” dating back to 1998, 
recommending changes that have not been legislated. 


¢ Department of Finance Canada staff are well-informed about the 
identified issues, but they do not make effective use of available 
electronic tools to record, track, and prioritize them for possible 
legislative amendment. 


- The Agency’s processes for tracking and monitoring issues in the Act 
are not adequate. The Agency has no formal database to keep track 
of the identified legislative issues and their disposition. Issues are 
validated, and analysis of some of them is carried out. 

However, there is no systematic review of their impact on 
compliance or on the tax base. As part of the annual Budget process, 
the Agency selects a relatively small number of outstanding issues as 
priorities for communication to the Department, based on the 
assessment of senior Agency officials. 


- The Agency provides guidance to taxpayers and tax practitioners in 
the form of advance income tax rulings, technical interpretations, 
Income Tax Interpretation Bulletins, as well as tax guides and 
pamphlets, to name a few. Tax practitioners we interviewed value the 
rulings process. However, in the past three years, the Agency has not 
met its timeliness standard for issuing advance income tax rulings. In 
addition, some of the information in the Agency’s Income Tax 
Interpretation Bulletins is no longer current and the public is not 
always made aware of that fact. 


The Agency and the Department have responded. The Agency and 
the Department agree with all of our recommendations. Their detailed 
responses follow each recommendation throughout the chapter. 
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What we examined 


Electronic Health Records 


Main Points 


Canada Health Infoway Inc. (Infoway) was created in 2001 as a 
federally funded, not-for-profit corporation to lead the national 
development of electronic health records (EHRs). Infoway’s goal is to 
ensure that, by 2010, every province and territory and the populations 
they serve will benefit from new health information systems that will 
help transform their health care system. Furthermore, Infoway’s stated 
goal is that 50 percent of Canadians will have their electronic health 
record available to their authorized health care professionals by 2010, 
and 100 percent by 2016. 


Health Canada transfers funds to Infoway based on funding 
agreements that specify the obligations of each party. Infoway has been 
granted $1.6 billion by the federal government. Infoway has allocated 
$1.2 billion to invest jointly with the provinces and territories in 
projects to accelerate the development and use of compatible EHRs 
across Canada, and $400 million for other related priorities. It is the 
role of the provinces and territories to propose the projects to Infoway 
for funding and to implement them. Thus, Infoway’s ability to achieve 
key outcomes in its funding agreements depends on the collaboration 
of the provinces and territories. 


We examined how Infoway manages funds from the federal 
government to achieve its goal of making compatible electronic health 
records available across Canada. In addition, we looked at the role of 
Health Canada, the sponsoring department, in ensuring that Infoway 
complies with the agreements under which it receives funding from the 
Department. For the most part, the audit focused on the fiscal 

years 2006-07 and 2007-08. Audit work for this chapter was 
substantially completed on 30 April 2009. 


Concurrent with our audit, six provincial audit offices audited how 
electronic health records funded by Infoway and/or provincial 
governments are being implemented in their respective provinces. The 
provincial audit offices will each report separately; a joint summary 
report on all of the audits will be issued in 2010. 
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Why it’s important 


What we found 


MAIN PoINTs—CHAPTERS 1 TO 8 


Electronic health records (EHRs) are intended to offer solutions to a 
number of persistent problems in Canada’s health system, some of 
which can be attributed to the use of paper-based health records. With 
EHRs, it is expected that health care professionals would be better able 
to share patient information, thus avoiding unnecessary or duplicate 
diagnostic tests, multiple prescriptions, and the risk of adverse drug 
reactions. Ultimately, the use of EHRs could reduce patient wait times, 
reduce costs, and save lives. From its inception in 2001 until 

31 March 2009, Infoway had spent $614.9 million on this initiative, 
and had committed an additional $614.2 million, for a total of 

$1.2 billion. Some experts have estimated the total cost of 
implementing EHRs Canada-wide at over $10 billion, and Infoway 
concurs. 


- Infoway has accomplished much in the eight years since its creation. 
Using the funding agreements with Health Canada as a starting 
point, Infoway developed an approach to providing for compatible 
electronic health records by identifying the key requirements and 
components of an EHR and developing a blueprint for the design of 
health information systems. It consulted widely with partners and 
stakeholders to obtain their input and support. In addition, it 
established appropriate governance mechanisms and developed a 
risk management strategy. It has implemented appropriate 
management controls for operational spending, although controls for 
contracting for goods and services need to be strengthened. 


In the 29 EHR projects we examined, Infoway had ensured that 
provinces and territories designed the projects to comply with 
requirements such as its blueprint and standards. It had also 
identified project-specific risks and was monitoring them, as well as 
other problems that arose during the life of the project. However, 
Infoway has not obtained the results of conformance testing on EHR 
systems. This means it does not have sufficient assurance that 
standards have been implemented as required. 


- Infoway has made considerable efforts to report on the progress of 
the EHR initiative. It reports progress toward its 2010 goal as the 
percentage of Canadians living in provinces or territories where an 
EHR is available to their health care professionals. However, it has 
not reported on other indicators of progress, such as the extent to 
which completed systems meet requirements for compatibility. Nor 
does it report on the adoption or use of completed systems by health 
care professionals, although it considers low adoption rates a serious 
risk to the EHR initiative. 


— 


Report of the Auditor General of Canada—Fall 2009 


Chapter 4 | 17 


MAIN Points—CHAPTERS 1 TO 8 


- As the sponsoring department, Health Canada periodically obtains 
assurance through audits and evaluations that Infoway is complying 
with the funding agreements. However, at the time of the audit, the 
Department still had not fully developed the monitoring framework 
it approved in 2008 to manage risks associated with such large 
amounts of funding and to strengthen ongoing monitoring of the 
Corporation, 


The entities have responded. The entities agree with all of our 
recommendations. Their detailed responses follow the 
recommendations throughout the chapter. 
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Chapter 5 


What we examined 


Why it’s important 


Acquiring Military Vehicles for Use 
in Afghanistan 


Main Points 


Canada’s military role in Afghanistan has been to contribute to 


‘international security and, in particular, to contribute to the stability of 


the area. National Defence has had to acquire new equipment urgently 
needed by the Canadian Forces to carry out their operations effectively 
and protect their personnel. 


We examined four urgent projects, each costing over $100 million, to 
acquire military vehicles that would improve operational capability 
and the protection of soldiers in Afghanistan. The projects involved 
replacement tanks, armoured patrol vehicles, armoured heavy support 
vehicles, and light armoured vehicles with remote weapon stations. 


We examined how National Defence managed the projects to ensure 
that the acquisitions met government policies related to project 
management and that the vehicles it was purchasing would meet the 
Canadian Forces’ urgent operational needs. We also examined how 
National Defence and Public Works and Government Services Canada 
(PWGSC) worked together to ensure that the contracting for the 
projects complied with government policies. 


In addition, we looked at the analysis and challenge role played by the 
Treasury Board of Canada Secretariat when project and contract 
proposals were submitted for Treasury Board approval. 


Audit work for this chapter was substantially Conte on 
29 May 2009. 


The Canadian Forces’ current deployment to Kandahar, Afghanistan has 
increased the risk to military and civilian personnel caused by threats 
such as improvised explosive devices. Failure to respond adequately and 
rapidly to these threats can have serious consequences—for example, 

it could leave the troops at greater risk or force Canada to adjust the 
scope of its military operations in Afghanistan. 


The ability to acquire new equipment rapidly is essential to help reduce 
the risks to personnel without compromising mission success. 
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What we found 


In three of the four projects we examined, National Defence and 
PWGSC provided the Canadian Forces in Afghanistan with urgently 
needed vehicles that National Defence determined met the 
operational needs. The quick procurement and delivery of these 
protective vehicles, in the opinion of National Defence, contributed 
to the safeguarding of Canadian soldiers in Afghanistan. National 
Defence and PWGSC successfully implemented some strategies to 
fast-track these three contracting processes, which potentially could 
be applied to streamlining non-urgent acquisitions. For the 

two competitive contracting processes that we examined, we found 
that both processes were managed in compliance with applicable 
contracting policies. However, we found some problems with roles 
and responsibilities and information provided to senior officials. 


The four projects we examined were not managed in accordance 
with National Defence’s project approval guide. Nor did the 
Department have a separate process in place, based for instance on 
the guide, to manage urgent acquisitions to ensure that the projects 
complied with government policies. Most of the required 
documentation was either not prepared or, if prepared, was deficient 
or prepared after the fact. Therefore, the documentation was of little 
value in helping to manage the projects or showing that the projects 
complied with government policies. 


While the Treasury Board of Canada Secretariat reviewed and 
challenged documents submitted by National Defence when it 
sought approval for the projects, some important issues were left 
unchallenged. For example, had officials asked why the required risk 
management strategies were not submitted, they would have learned 
that the strategies had not been prepared—a signal that the project 
management policies may not have been followed. For its part, 
National Defence did not disclose some important information 
needed for informed decision making, which made it difficult 

for the Secretariat to fulfill its challenge function. For instance, 
National Defence did not state that there was a high risk that it 
would not be able to replace all the old tanks in Afghanistan with 
the new Leopard 2 tanks, which was one of the project’s goals. 


The entities have responded. The entities agree with all of our 


recommendations. Their detailed responses follow the 
recommendations throughout the chapter. 
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- What we examined 


Why it’s important 


Land Management and 
Environmental Protection on Reserves 


Main Points 


Land management refers to managing the use and development of land 
resources in a sustainable way. Land can be used for a range of purposes 
that interact and may compete with one another, making it desirable 
to plan and manage land use in an integrated way that provides for 
protecting the environment from negative impacts of economic 
development. 


Until the early 1980s, all land management activities on First Nations 
reserves were the exclusive responsibility of the federal government. 
However, through comprehensive land claim settlements and 
self-government agreements, several First Nations have assumed full 
control and responsibility for the development of the reserve lands; 
some others have assumed responsibility under other arrangements 
for various aspects of land management on their reserves. The specific 
roles and responsibilities of the federal government vary depending 
on its agreement or arrangements with the First Nation. 


We examined how Indian and Northern Affairs Canada (INAC) 

and Environment Canada have carried out the federal government’s 
responsibilities for land management and environmental protection on 
reserves. This included looking at regulatory and non-regulatory 
measures used to manage the environment and the support INAC 
provides to those First Nations wishing to assume more land 
management responsibilities. 


Audit work for this chapter was substantially completed on 


29 May 2009. 


Reserve lands are central to First Nations peoples’ history, cultural 
identity, and day-to-day activities. Many First Nations are among 
the most economically deprived communities in the country. Their 
sustainable economic development depends on their access to and 
control over their land and natural resources and on a clean and 
healthy environment. Without the capacity and means to develop 
and use their lands and resources sustainably for their economic 
benefit, the opportunities for First Nations to improve their quality 
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of life and approach the standard of health and well-being enjoyed by 
other communities in Canada are severely restricted. 


What we found = -_—Indian and Northern Affairs Canada and Environment Canada have 
identified a significant -gap between First Nations reserves and 
Canadian communities elsewhere in the degree to which regulations 
protect the environment. While the federal government has the 
authority to regulate environmental threats on reserves, it has rarely 
used this authority to develop regulations to mitigate environmental 
threats that are regulated off reserves by provincial governments. 


* INAC has done little to monitor and enforce compliance with 
existing regulations. For example, while regulations under the 
Indian Act require a permit issued by INAC to operate a landfill site 
or burn waste on reserve lands, the Department has issued few 
permits and is not equipped to conduct inspections, monitor 
compliance, and enforce the regulations. Consequently, garbage is 
often not confined to licensed landfill sites and there is no 
monitoring of the impacts on drinking water sources and air quality. 
Off reserves, provincial and municipal regulations and enforcement 
help to prevent such situations. 


Although INAC has developed legislative and program options to 
support First Nations who wish to assume partial or full control of 
land management on their reserves, most First Nations lands are still 
managed by the Department under the Indian Act. First Nations’ 
access to alternative land management regimes established by INAC 
does not meet the demand. Consequently, INAC is.unable to keep 
its commitment to transfer greater control of land management to 
First Nations who want it and are ready to take on these 
responsibilities. Furthermore, for First Nations under either of the 
alternative land management regimes, the Department provides 
insufficient training in comparison with the land management 
responsibilities it is transferring to them. 


¢ Officials of both INAC and Environment Canada cited a lack of 


funding as a key reason for not meeting some of their commitments. 


The departments have responded. The departments agree with all of 
our recommendations. Their detailed responses follow each 
recommendation throughout the chapter. 
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What we examined 


Emergency Management— 


Public Safety Canada 


Main Points 


Emergency management refers to a wide range of measures to protect 
communities and the environment from risks and to recover from 
emergency events stemming from either natural or human-induced 
causes. While some emergencies in Canada can be handled locally by 


municipalities or provinces, the federal government will assist when 


requested, when the emergency transcends jurisdictional boundaries, 
or when its assistance is in the national interest. As emergency events 
today can escalate quickly, this federal capability has become 
increasingly necessary. 


Through legislation and government policy, Public Safety Canada, 
which was created in December 2003, is responsible for leading by 
coordinating the management of emergencies among federal 
departments and agencies. This includes establishing policies and 
programs for the preparation, testing and exercising, and implementing 
emergency management plans; it also includes monitoring and 
coordinating a common federal approach to emergency response along 
with the provinces—an “all-hazards” approach incorporating 
prevention and mitigation, preparedness, response, and recovery. The 
Department’s responsibility for emergency management includes 
coordinating the protection of critical infrastructure—from planning 
for emergencies to recovering from them. Critical infrastructure 
includes physical and information technology facilities, networks, 
services, and assets essential to the health and safety or economic 
well-being of Canadians. 


We examined how Public Safety Canada carries out these 
responsibilities. In addition, we looked at its efforts to enhance 
emergency response and recovery in coordination with six other 
departments that have specific roles in emergency management. Our 
audit included assessing the government’s progress on some of the 
commitments it made to Parliament. Our audit covers performance of 
federal departments and agencies and events taking place since our 
last audit, reported in April 2005, and 15 June 2009. 
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Why it’s important 


What we found 


We did not examine the performance of emergency management 
efforts by provinces, territories, or local communities. 


The HINI1 pandemic, the 2003 eastern seaboard power blackout, 
Severe Acute Respiratory Syndrome (SARS), massive flooding, and 
terrorist conspiracies and attacks have demonstrated that global trade, 
international travel, and cyberspace have increased the speed at which 
emergencies escalate in scope and severity. Today, many emergencies 
can be difficult to contain by a single government department or 
jurisdiction. A federal response is needed for emergencies that are 
beyond the capacities of other players—emergencies that may have a 
low probability of occurrence but a high potential impact. 


Public Safety Canada is faced with the challenging task of providing 
the coordination necessary for an overall federal approach to 
emergency management, in an environment where departments have 
operated as needed and through their ministers to provide federal 
assistance on a case-by-case basis. 


« Public Safety Canada has not exercised the leadership necessary to 
coordinate emergency management activities, including critical 
infrastructure protection in Canada. For example, it has yet to 
develop the policies and programs that would help clarify its 
leadership and coordination role for an “all-hazards” approach to the 
emergency management activities of departments. Public Safety 
Canada has taken the first step by developing the interim Federal 
Emergency Response Plan, which it considers to be final although it 
has not been formally approved by the government. Nor does the 
Plan include updated or completed definitions of the roles, 
responsibilities, and capabilities needed for an integrated, 
coordinated approach to emergency response. 


Public Safety Canada has made considerable progress in improving 
federal emergency coordination through its Government Operations 
Centre. It keeps other departments informed of the status of events 
on a real-time basis and also produces regular situation awareness 
reports for such issues as the H1IN1 virus, which allows decisions to 
be based on a common set of facts. 


* Public Safety Canada has developed a strategy to protect Canada’s 
critical infrastructure, but this strategy is still in draft form. At the 
time of our audit, the critical infrastructure that needs to be 
protected had not yet been determined. Public Safety has moved 
forward in promoting a consistent approach to protection efforts 
across government. For example, it has categorized critical 
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infrastructure into 10 sectors, each headed by a federal department. 
However, it has not provided those departments with guidance for 
determining what assets or facilities are critical and require 
protection. 


- Progress has been slow until 2009 on Public Safety Canada’s 2004 
commitment to develop a cyber security strategy, although threats to 
computer-based critical infrastructure, including federal information 
systems, have been growing and evolving. To date, it has identified 
the key elements of a cyber strategy and initiated action on a list of 
current cyber security initiatives along with other federal 
government departments. However, at the time of our audit, no 
date was planned for obtaining formal approval of the strategy. 


« Although the 2004 National Security Policy called for first 
responders’ equipment and communications to be interoperable, 
key gaps remain for voice communications. This limits the ability of 
fire, police, and ambulance services to work together and with other 
jurisdictions in an emergency. The Department has directed little or 
no funding toward standardizing equipment. 


The Department and the Privy Council Office have responded. The 
Department and the Privy Council Office agree with all of the 
recommendations that are addressed to them. Their detailed responses 
follow the recommendations throughout the chapter. 
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Chapter 8 


What we examined 


Why it’s important 


Strengthening Aid Effectiveness— 
Canadian International Development 
Agency 7 


Main Points 


The Canadian International Development Agency (CIDA) 
administers the bulk of Canada’s official development assistance 
program. Its mission is to lead Canada’s international effort to help 
people living in poverty. It does not have governing legislation that 
defines its mandate and role. 


In its 2002 Policy Statement on Strengthening Aid Effectiveness, 
CIDA committed to align its efforts with recipient countries’ needs 
and priorities; harmonize its activities with those of other donors; and 
use new forms of aid known as program-based approaches, in which 
donors coordinate support to the budgets of recipient governments or 
local organizations for a development program delivered using local 
systems and procedures. Recognizing that its programming is widely 
dispersed, CIDA has also committed to focus its aid in fewer sectors in 
order to make a more meaningful contribution. 


We examined how the Agency is implementing these commitments. 
We looked at management processes in place at its head office to guide 
and sustain the implementation of the commitments. In addition, we 
selected five countries and looked at how CIDA plans its activities 
there, selects projects for funding, and monitors project risks. 


Audit work for this chapter was substantially completed 15 May 2009. 


Development assistance is a key means to help developing countries 
reduce poverty. It is estimated that over 1.4 billion people are living on 
less than US$1.25 a day—the latest international poverty line set by 
the World Bank. CIDA’s commitments reflect principles of aid 
effectiveness that international donor countries have recognized will 
lead to more sustainable and self-reliant development. 


By its nature, international development requires a long-term effort 
and stable, predictable programming. CIDA has declared aid 
effectiveness to be its overarching priority, recognizing that this 
involves risks of a different nature and level and significantly changes 
the way it needs to work with other donors and recipient governments. 
Robust management processes—which include specific action plans, 
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performance targets, assigned accountabilities, and management 
review—are crucial to sustain implementation of its commitments and 
directions over time. 


What we found - In the field, CIDA is working in collaboration with other donors to 
" align its development assistance with the needs of the recipient 

countries we looked at. It is working to harmonize its efforts with 
those of other donors and to use program-based approaches where 
appropriate. Its selection of projects for funding is based on a 
rationale consistent with principles of aid effectiveness. We were told 
that CIDA is well regarded in the field and that its partners 
appreciate its efforts. However, the complex and lengthy processes 
required to obtain corporate approval for project funding have long 
been criticized within and outside the Agency. 


- Corporate support in the Agency for program-based approaches has 
been neither uniform nor timely. In the early years after the 2002 
Policy Statement was released, management provided little specific 
direction to field operations on how and when to use program-based 
approaches, and there are no quantitative targets for the extent of 
their use. The Agency’s internal processes do not clearly stipulate 
the conditions under which it would or would not participate in a 
program-based approach or the types of risk assessment to be done 
before agreeing to participate. However, it had undertaken some 
form of risk assessment for the program-based approaches we 
examined or it had relied on its donor partners’ analyses. It also 
monitored project risks. 


* CIDA’s main planning documents—Country Development 
Programming Frameworks—for the countries we examined have 
expired, and the process used to prepare and maintain them has been 
discontinued. The Agency is developing a new planning process but, 
by the end of our audit, the status of the new plans was unclear 
because none had received formal approval or been communicated 
broadly within CIDA. None have been made public. This means that 
donors, recipient governments, and program staff are unclear about 
the Agency’s direction and long-term commitment to specific 


countries or regions. 


- As a result of lack of clear direction and action plans, coupled with 
broadly defined and shifting priority sectors, CIDA is not realizing its 
goal of making a more meaningful Canadian contribution in a 
country or region by focusing its aid more narrowly. It has made 
limited progress in determining its strengths relative to other donors 
as well as the type of expertise it requires to support its priority 
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sectors and how best to provide it. The lack of direction and shifting 
priorities have undermined the long-term predictability of the 
Agency’s efforts. 


Many of the weaknesses we identified can be traced to a lack of 
corporate management processes to guide and monitor the 
implementation of CIDA’s aid effectiveness commitments. The 
Agency’s early intentions were not matched with specific action 
plans and follow-through, and it has yet to develop a comprehensive 
strategy for implementing its commitments. In addition, frequent 
changes in policy direction and substantial turnover of senior 
personnel in recent years have posed significant challenges for 
CIDA’s aid effectiveness agenda. 


The Agency has responded. The Agency agrees with all of our 
recommendations. Its detailed responses follow the recommendations 
throughout the chapter. 
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APPENDICES 


Appendix A Auditor General Act 


Short title 


Definitions 


“appropriate Minister” 


“Auditor General” 


“category I 
department” 


“Commissioner” 


“Crown corporation” 


- “department” 


“funding agreement” 


“recipient” 


“registrar” 


Short Title 
1. This Act may be cited as the Auditor General Act. 
Interpretation 


2y In this Act, 


“appropriate Minister” has the meaning assigned by section 2 of the Financial 
Administration Act; 


“Auditor General” means the Auditor General of Canada appointed pursuant to 
subsection 3(1); 


“category I department” means 


(a) any department named in schedule I to the Financial 
Administration Act, 


(b) any department in respect of which a direction has been made 
under subsection 11(3) of the Federal Sustainable Development Act; 
and 


(c) any agency set out in the schedule to the Federal Sustainable 
Development Act. 


“Commissioner” means the Commissioner of the Environment and Sustainable 
Development appointed under subsection 15.1(1); 


“Crown corporation” has the meaning assigned to that expression by section 83 of 
the Financial Administration Act; 


“department” has the meaning assigned to that term by section 2 of the Financial 
Administration Act; 


“funding agreement” has the meaning given to that expression by subsection 


42 (4) of the Financial Administration Act; 


“recipient” has the meaning given to that expression by subsection 42 (4) of the 
Financial Administration Act; 


“registrar” means the Bank of Canada and a registrar appointed under Part IV of 
the Financial Administration Act; 
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“sustainable 
development” 


Control 


Control 


Appointment 


‘Tenure 


Ceasing to hold office 


Re-appointment 


Interim appointment 


“sustainable development” means development that meets the needs of the 
present without compromising the ability of future generations to meet their own 


needs; 


2.1 (1) For the purpose of paragraph (d) of the definition “recipient” in 
subsection 42 (4) of the Financial Administration Act, a municipality or 
government controls.a corporation with share capital if 


(a) shares of the corporation to which are attached more than fifty per 
cent of the votes that may be cast to elect directors of the 
corporation are held, otherwise than by way of security only, by, on 
behalf of or in trust for that municipality or government; and 


(b) the votes attached to those shares are sufficient, if exercised, to 
elect a majority of the directors of the corporation. 


(2) For the purpose of paragraph (d) of the definition “recipient” in 
subsection 42(4) of the Financial Administration Act, a corporation without share 
capital is controlled by a municipality or government if it is able to appoint the 
majority of the directors of the corporation, whether or not it does so. 


Auditor General of Canada 


3. (1) The Governor in Council shall, by commission under the Great Seal, 
appoint an Auditor General of Canada after consultation with the leader of every 
recognized party in the Senate and House of Commons and approval of the 
appointment by resolution of the Senate and House of Commons. 


(1.1) The Auditor General holds office during good behaviour for a term of 
10 years but may be removed for cause by the Governor in Council on address of 
the Senate and House of Commons. 


(2) Despite subsections (1) and (1.1), the Auditor General ceases to hold 
office on reaching 65 years of age. 


(3) Once having served as the Auditor General, a person is not eligible for 
re-appointment to that office. 


(4) In the event of the absence or incapacity of the Auditor General or if 
that office is vacant, the Governor in Council may appoint any qualified auditor 
to hold that office in the interim for a term not exceeding six months, and that 
person shall, while holding office, be paid the salary or other remuneration and 
expenses that may be fixed by the Governor in Council. 
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Salary 4. (1) The Auditor General shall be paid a salary equal to the salary of a 
puisne judge of the Supreme Court of Canada. 


Pension benefits (2) The provisions of the Public Service Superannuation Act, other than 
those relating to tenure of office, apply to the Auditor General except that a 
person appointed as Auditor General from outside the public service may, by 
notice in writing given to the President of the Treasury Board not more than sixty 
days after the date of his appointment as Auditor General, elect to participate in 
the pension plan provided for in the Diplomatic Service (Special) Superannuation 
Act in which case the provisions of that Act, other than those relating to tenure 
of office, apply to him and the provisions of the Public Service Superannuation Act 
do not apply to him. 


Powers and Duties 


Examination 5. The Auditor General is the auditor of the accounts of Canada, including 
those relating to the Consolidated Revenue Fund and as such shall make such 
examinations and inquiries as he considers necessary to enable him to report as 
required by this Act. | 


Idem 6. The Auditor General shall examine the several financial statements 
required by section 64 of the Financial Administration Act to be included in the 
Public Accounts, and any other statement that the President of the Treasury 
Board or the Minister of Finance may present for audit and shall express his 
opinion as to whether they present fairly information in accordance with stated 
accounting policies of the federal government and on a basis consistent with that 
of the preceding year together with any reservations he may have. 


Annual and additional 7. (1) The Auditor General shall report annually to the House of Commons 

reports to the House — and may make, in addition to any special report made under subsection 8(1) or 

eh Commons 19(2) and the Commissioner’s report under subsection 23(2), not more than 
three additional reports in any year to the House of Commons 


(a) on the work of his office; and, 


(b) on whether, in carrying on the work of his office, he received all the 
information and explanations he required. 
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Idem 


Submission of annual 
report to Speaker and 
tabling in the House 
of Commons 


Notice of additional 
reports to Speaker and 
tabling in the House 
of Commons 


Submission of 
additional reports to 
Speaker and tabling 
in the House of 
Commons 


(2) Each report of the Auditor General under subsection (1) shall call 
attention to anything that he considers to be of significance and of a nature that 
should be brought to the attention of the House of Commons, including any 
cases in which he has observed that 


(a) accounts have not been faithfully and properly maintained or 
public money has not been fully accounted for or paid, where so 
required by law, into the Consolidated Revenue Fund; 


(b) essential records have not been maintained or the rules and 
procedures applied have been insufficient to safeguard and control 
public property, to secure an effective check on the assessment, 
collection and proper allocation of the revenue and to ensure that 
expenditures have been made only as authorized; 


(c) money has been expended other than for purposes for which it was 
appropriated by Parliament; 


(d) money has been expended without due regard to economy or 
efficiency; 


(e) satisfactory procedures have not been established to measure and 
report the effectiveness of programs, where such procedures could 
appropriately and reasonably be implemented; or 


(f) money has been expended without due regard to the 
environmental effects of those expenditures in the context of 
sustainable development. 


(3) Each annual report by the Auditor General to the House of Commons 
shall be submitted to the Speaker of the House of Commons on or before 
December 31 in the year to which the report relates and the Speaker of the 
House of Commons shall lay each such report before the House of Commons 
forthwith after receiving it or, if that House is not then sitting, on any of the first 
fifteen days on which that House is sitting after the Speaker receives it. 


(4) Where the Auditor General proposes to make an additional report 
under subsection (1), the Auditor General shall send written notice to the 
Speaker of the House of Commons of the subject-matter of the proposed report. 


(5) Each additional report of the Auditor General to the House of 
Commons made under subsection (1) shall be submitted to the House of 
Commons on the expiration of thirty days after the notice is sent pursuant to 
subsection (4) or any longer period that is specified in the notice and the Speaker 
of the House of Commons shall lay each such report before the House of 
Commons forthwith after receiving it or, if that House is not then sitting, on any 
of the first fifteen days on which that House is sitting after the Speaker receives it. 
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Inquiry and report 7.1 (1) The Auditor General may, with respect to a recipient under any funding 
agreement, inquire into whether 


(a) the recipient has failed to fulfil its obligations under any funding 
agreement; 


(b) money the recipient has received under any funding agreement has 
been used without due regard to economy and efficiency; 


(c) the recipient has failed to establish satisfactory procedures to 
measure and report on the effectiveness of its activities in relation 
to the objectives for which it received funding under any funding 
agreement; 


(d) the recipient has failed to faithfully and properly maintain accounts 
and essential records in relation to any amount it has received 
under any funding agreement; or 


(e) money the recipient has received under any funding agreement has 
been expended without due regard to the environmental effects of 
those expenditures in the context of sustainable development. 


Report (2) The Auditor General may set out his or her conclusions in respect of an 
inquiry into any matter referred to in subsection (1) in the annual report, or in 
any of the three additional reports, referred to in subsection 7(1). The Auditor 
General may also set out in that report anything emerging from the inquiry that 
he or she considers to be of significance and of a nature that should be brought to 
the attention of the House of Commons. 


Special report tothe 8. (1) The Auditor General may make a special report to the House of 

House of Commons = Commons on any matter of pressing importance or urgency that, in the opinion 
of the Auditor General, should not be deferred until the presentation of the next 
report under subsection 7(1). 


Submission of reports (2) Each special report of the Auditor General to the House of Commons 

= Speaker and tabling jyade under subsection (1) or 19(2) shall be submitted to the Speaker of the 

pee ae House of Commons and shall be laid before the House of Commons by the 
Speaker of the House of Commons forthwith after receipt thereof by him, or if 
that House is not then sitting, on the first day next thereafter that the House of 


Commons is sitting. 


LE = See : —--« 
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Improper retention of 
public money 


Inquiry and report 


Advisory powers 


Access to information 


Stationing of officers 
in departments 


9. The Auditor General shall 


(a) make such examination of the accounts and records of each 
registrar as he deems necessary, and such other examinations of a 
registrar’s transactions as the Minister of Finance may require, and 


(b) when and to the extent required by the Minister of Finance, 
participate in the destruction of any redeemed or cancelled 
securities or unissued reserves of securities authorized to be 
destroyed under the Financial Administration Act, 


and he may, by arrangement with a registrar, maintain custody and control, 
jointly with that registrar, of cancelled and unissued securities. 


10. Whenever it appears to the Auditor General that any public money has 
been improperly retained by any person, he shall forthwith report the 
circumstances of the case to the President of the Treasury Board. 


11. The Auditor General may, if in his opinion such an assignment does not 
interfere with his primary responsibilities, whenever the Governor in Council so 
requests, inquire into and report on any matter relating to the financial affairs of 
Canada or to public property or inquire into and report on any person or 
organization that has received financial aid from the Government of Canada or in 
respect of which financial aid from the Government of Canada is sought. 


12. The Auditor General may advise appropriate officers and employees in the 
federal public administration of matters discovered in his examinations and, in 
particular, may draw any such matter to the attention of officers and employees 
engaged in the conduct of the business of the Treasury Board. 


Access to Information 


13. (1) Except as provided by any other Act of Parliament that expressly refers 
to this subsection, the Auditor General is entitled to free access at all convenient 
times to information that relates to the fulfilment of his or her responsibilities and 
he or she is also entitled to require and receive from members of the federal 
public administration such information, reports and explanations as he or she 
considers necessary for that purpose. 


(2) In order to carry out his duties more effectively, the Auditor General 
may station in any department any person employed in his office, and the 


department shall provide the necessary office accommodation for any person so 
stationed. 
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(3) The Auditor General shall require every person employed in his office 
who is to examine the accounts of a department or of a Crown corporation 
pursuant to this Act to comply with any security requirements applicable to, and 
to take any oath of secrecy required to be taken by, persons employed in that 
department or Crown corporation. 


(4) The Auditor General may examine any person on oath on any matter 
pertaining to any account subject to audit by him and for the purposes of any 
such: examination the Auditor General may exercise all the powers of a 
commissioner under Part I of the Inquiries Act. 


14. (1) Notwithstanding subsections (2) and (3), in order to fulfil his 
responsibilities as the auditor of the accounts of Canada, the Auditor General 
may rely on the report of the duly appointed auditor of a Crown corporation or of 
any subsidiary of a Crown corporation. 


(2) The Auditor General may request a Crown corporation to obtain and 
furnish him such information and explanations from its present or former 
directors, officers, employees, agents, and auditors or those of any of its 
subsidiaries as are, in his opinion, necessary to enable him to fulfil his 
responsibilities as the auditor of the accounts of Canada. 


(3) If, in the opinion of the Auditor General, a Crown corporation, in 
response to a request made under subsection (2), fails to provide any or sufficient 
information or explanations, he may so advise the Governor in Council, who may 
thereupon direct the officers of the corporation to furnish the Auditor General 
with such information and explanations and to give him access to those records, 
documents, books, accounts and vouchers of the corporation or any of its 
subsidiaries access to which is, in the opinion of the Auditor General, necessary 
for him to fulfil his responsibilities as the auditor of the accounts of Canada. 


Staff of the Auditor General 


15. (1) The officers and employees that are necessary to enable the Auditor 
General to perform his or her duties are to be appointed in accordance with the 
Public Service Employment Act and, subject to subsections (2) to (5), the 
provisions of that Act apply to those officers and employees. 


(2) The Auditor General may exercise the powers and perform the 
functions of the employer and deputy head under the Public Service Employment 
Act within the meaning of those terms in subsection 2(1) of that Act. 
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(3) The Auditor General may, in the manner and subject to the terms and 
conditions that the Public Service Commission directs, exercise the powers and 
perform the functions of that Commission under the Public Service Employment 
Act, other than its powers and functions in relation to the hearing of allegations 
by a candidate under sections 118 and 119 of that Act and its power to make 


regulations. 


(4) The Auditor General may authorize any person employed in his or her 
office to exercise and perform, in any manner and subject to any terms and 
conditions that he or she directs, any of his or her powers and functions under 
subsections (2) and (3). 


(5) Any person authorized under subsection (4) may, subject to and in 
accordance with the authorization, authorize one or more persons under that 
person’s jurisdiction to exercise any power or perform any function to which the 
authorization relates. 


15.1 (1) The Auditor General shall, in accordance with the Public Service 
Employment Act, appoint a senior officer to be called the Commissioner of the 
Environment and Sustainable Development who shall report directly to the 
Auditor General. 


(2) The Commissioner shall assist the Auditor General in performing the 
duties of the Auditor General set out in this Act that relate to the environment 
and sustainable development. 


16. The Auditor General is authorized, in respect of persons appointed in his or 
her office, to exercise the powers and perform the functions of the Treasury Board 
that relate to human resources management within the meaning of paragraph 
7(1)(e) and section 11.1 of the Financial Administration Act, as well as those of 
deputy heads under subsection 12(2) of that Act, as that subsection reads 
without regard to any terms and conditions that the Governor in Council may 
direct, including the determination of terms and conditions of employment and 
the responsibility for employer and employee relations. 


16.1 (1) The Auditor General may authorize any person employed in his or her 
office to exercise and perform, in any manner and subject to any terms and 
conditions that he or she directs, any of his or her powers and functions in 
relation to human resources management. 


(2) Any person authorized under subsection (1) may, subject to and in 
aecordance with the authorization, authorize one or more persons under that 


; . ° ‘ . 7 F . 
person’s jurisdiction to exercise any power or perform any function to which the 
authorization relates. 
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16.2 Subject to any other Act of Parliament or regulations made under any Act 
of Parliament, but without the approval of the Treasury Board, the Auditor 
General may, within the total dollar limitations established for his or her office in 
appropriation Acts, contract for professional services. 


17. Classification standards may be prepared for persons employed in the office 
of the Auditor General to conform with the classifications that the Auditor 
General recognizes for the purposes of that office. 


18. The Auditor General may designate a senior member of his staff to sign on 
his behalf any opinion that he is required to give and any report, other than his 
annual report on the financial statements of Canada made pursuant to section 64 
of the Financial Administration Act and his reports to the House of Commons 
under this Act, and any member so signing an opinion or report shall indicate 
beneath his signature his position in the office of the Auditor General and the 
fact that he is signing on behalf of the Auditor General. 


Immunities 


18.1 The Auditor General, or any person acting on behalf or under the direction 
of the Auditor General, is not a competent or compellable witness — in respect 
of any matter coming to the knowledge of the Auditor General or that person as a 
result of performing audit powers, duties or functions under this or any other Act 
of Parliament during an examination or inquiry — in any proceedings other than 
a prosecution for an offence under section 131 of the Criminal Code (perjury) in 
respect of a statement made under this Act. 


18.2 (1) No criminal or civil proceedings lie against the Auditor General, or 
against any person acting on behalf or under the direction of the Auditor 
General, for anything done, reported or said in good faith in the course of the 
performance or purported performance of audit powers, duties or functions under 
this or any other Act of Parliament. 


(2) For the purposes of any law relating to defamation, 


(a) anything said, any information supplied or any document or thing 
produced in good faith by or on behalf of the Auditor General, in 
the course of the performance or purported performance of audit 
powers, duties or functions under this or any other Act of 
Parliament, is privileged; and 


(b) any report made in good faith by the Auditor General in the course 
of the performance or purported performance of audit powers, 
duties or functions under this or any other Act of Parliament, and 
any fair and accurate account of the report made in good faith in a 
newspaper or any other periodical publication or in a broadcast, is 


privileged. 
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19. (1) The Auditor General shall annually prepare an estimate of the sums 
that will be required to be provided by Parliament for the payment of the salaries, 
allowances and expenses of his office during the next ensuing fiscal year. 


(2) The Auditor General may make a special report to the House of 
Commons in the event that amounts provided for his office in the estimates 
submitted to Parliament are, in his opinion, inadequate to enable him to fulfil the 
responsibilities of his office. 


20. The provisions of the Financial Administration Act with respect to the 
division of appropriations into allotments do not apply in respect of 
appropriations for the office of the Auditor General. 


Audit of the Office of the Auditor General 


21. (1) A qualified auditor nominated by the Treasury Board shall examine the 
receipts and disbursements of the office of the Auditor General and shall report 
annually the outcome of his examinations to the House of Commons. 


(2) Each report referred to in subsection (1) shall be submitted to the 
President of the Treasury Board on or before the 31st day of December in the year 
to which the report relates and the President of the Treasury Board shall lay each 
such report before the House of Commons within fifteen days after receipt 
thereof by him or, if that House is not then sitting, on any of the first fifteen days 
next thereafter that the House of Commons is sitting. 


Sustainable Development 


21.1 In addition to carrying out the functions referred to in subsections 23 (3) 
and (4), the purpose of the Commissioner is to provide sustainable development 
monitoring and reporting on the progress of category I departments towards 
sustainable development, which is a continually evolving concept based on the 
integration of social, economic and environmental concerns, and which may be 
achieved by, among other things, 


) the integration of the environment and the economy; 
) protecting the health of Canadians; 

c) protecting ecosystems; 

) meeting international obligations; 

) promoting equity; 


f) an integrated approach to planning and making decisions that 
takes into account the environmental and natural resource costs of 
different economic options and the economic costs of different 
environmental and natural resource options; 
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(g) preventing pollution; and 


(h) respect for nature and the needs of future generations. 


22. (1) Where the Auditor General receives a petition in writing from a 
resident of Canada about an environmental matter in the context of sustainable 
development that is the responsibility of a category I department, the Auditor 
General shall make a record of the petition and forward the petition within 
fifteen days after the day on which it is received to the appropriate Minister for 
the department. 


(2) Within fifteen days after the day on which the Minister receives the 
petition from the Auditor General, the Minister shall send to the person who 
made the petition an acknowledgement of receipt of the petition and shall send a 
copy of the acknowledgement to the Auditor General. 


(3) The Minister shall consider the petition and send to the person who 
made it a reply that responds to it, and shall send a copy of the reply to the 
Auditor General, within 


(a) one hundred and twenty days after the day on which the Minister 
receives the petition from the Auditor General; or 


(b) any longer time, where the Minister personally, within those one 
hundred and twenty days, notifies the person who made the 
petition that it is not possible to reply within those one hundred 
and twenty days and sends a copy of that notification to the 
Auditor General. 


(4) Where the petition is from more than one person, it is sufficient for the 
Minister to send the acknowledgement and reply, and the notification, if any, to 
one or more of the petitioners rather than to all of them. 


23. (1) The Commissioner shall make any examinations and inquiries that the 
Commissioner considers necessary in order to monitor 


(a) the extent to which category I departments have contributed to 
meeting the targets set out in the Federal Sustainable Development 
Strategy and have met the objectives, and implemented the plans, 
set out in their own sustainable development strategies laid before 
the House of Commons under section 11 of the Federal Sustainable 
Development Act; and 


(b) the replies by Ministers required by subsection 22(3). 
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Commissioner’s report (2) The Commissioner shall, on behalf of the Auditor General, report 
annually to the House of Commons concerning anything that the Commissioner 
considers should be brought to the attention of that House in relation to 
environmental and other aspects of sustainable development, including 


(a) the extent to which category I departments have contributed to 
meeting the targets set out in the Federal Sustainable Development 
Strategy and have met the objectives, and implemented the plans, set 
out in their own sustainable development strategies laid before that 
House under section 11 of the Federal Sustainable Development Act; 


(b) the number of petitions recorded as required by subsection 22(1), 
the subject-matter of the petitions and their status; and 


(c) the exercising of the authority of the Governor in Council under 
subsections 11(3) and (4) of the Federal Sustainable Development Act. 


Duty to examine (3) The Commissioner shall examine the report required under 
subsection 7(2) of the Federal Sustainable Development Act in order to assess the 
fairness of the information contained in the report with respect to the progress of 
the federal government in implementing the Federal Sustainable Development 
Strategy and meeting its targets. 


Duty to report (4) The Commissioner shall include in the report referred to in 
subsection (2) the results of any assessment conducted under subsection (3) since 
the last report was laid before the House of Commons under subsection (5). 


Submission and tabling (5) The report required by subsection (2) shall be submitted to the Speaker 

of report of the House of Commons and shall be laid before that House by the Speaker on 
any of the next 15 days on which that House is sitting after the Speaker receives 
it. 
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Appendix B Reports of the Standing Committee on Public Accounts to the House of Commons, 2008-09 


The following reports have been tabled since our December 2008 Report went to print. They are available 
on the website of Canada’s Parliament (www.parl.gc.ca). 


40th Parliament, 2nd Session 


Report 1—Chapter 4, Military Health Care—National Defence, of the October 2007 Report of the 
Auditor General of Canada (Adopted by the Committee on 12 February 2009; presented to the House 
on 25 February 2009) | 


Report 2—Chapter 5, Keeping the Border Open and Secure—Canada Border Services Agency, of the 
October 2007 Report of the Auditor General of Canada (Adopted by the Committee on 12 February 2009; 
presented to the House on 25 February 2009) 


Report 3—Chapter 3, Inuvialuit Final Agreement, of the October 2007 Report of the Auditor General 
of Canada (Adopted by the Committee on 12 February 2009; presented to the House on 25 February 2009) 


Report 4—Chapter 5, Managing the Delivery of Legal Services to Government—Department of Justice 
Canada, of the May 2007 Report of the Auditor General of Canada (Adopted by the Committee on 
12 February 2009; presented to the House on 25 February 2009) 


Report 5—Chapter 4, Canadian Agricultural Income Stabilization—Agriculture and Agri-Food Gln 
of the May 2007 Report of the Auditor General of Canada (Adopted by the Committee on 12 February 
2009; presented to the House on 25 February 2009) 


Report 6—Public Accounts of Canada 2008 (Adopted by the Committee on 5 March 2009; presented 
to the House on 24 March 2009) 


Report 7—Chapter 4, First Nations Child and Family Services Program—Indian and Northern Affairs 
Canada, of the May 2008 Report of the Auditor General of Canada (Adopted by the Committee on 
5 March 2009; presented to the House on 24 March 2009) 


Report 8—Main Estimates 2009-2010: Vote 15 under Finance, and Part I]]—Report on Plans and 
Priorities and Departmental Performance Report of the Office of the Auditor General of Canada (Adopted 
by the Committee on 23 April 2009; presented to the House on 13 May 2009) 


Report 9—Chapter 7, Detention and Removal of Individuals—Canada Border Services Agency, 
of the May 2008 Report of the Auditor General of Canada (Adopted by the Committee on 28 April 2009; 
presented to the House on 13 May 2009) 


Report 10—Chapter 1, Management of Fees in Selected Departments and Agencies, of the May 2008 
Report of the Auditor General of Canada (Adopted by the Committee on 28 April 2009; presented to 
the House on 13 May 2009) 


Report 11—Chapter 1, A Study of Federal Transfers to Provinces and Territories, of the December 2008 
Report of the Auditor General of Canada (Adopted by the Committee on 7 May 2009; presented to 
the House on 27 May 2009) 
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Report 12—Chapter 5, Surveillance of Infectious Diseases—Public Health Agency of Canada, 
of the May 2008 Report of the Auditor General of Canada genes by the Committee on 7 May 2009; 
presented to the House on 27 May 2009) 


Report 13—Supplementary Estimates (A) 2009-2010: Vote 15a under Finance (Adopted by the 
Committee on 26 May 2009; presented to the House on 10 June 2009) 


Report 14—Chapter 2, Governance of Small. Federal Entities, of the December 2008 Report of the 
Auditor General of Canada (Adopted by the Committee on 28 May 2009; presented to the House on 
10 June 2009) 


Report 15—Chapter 7, Economy and Efficiency of Services—Correctional Service Canada, of the 
December 2008 Report of the Auditor General of Canada (Adopted by the Committee on 4 June 2009; 
presented to the House on 16 June 2009) 


Report 16—Main Estimates 2009-2010: Vote 15 under Finance, and Part I]]—Report ‘on Plans and 
Priorities and Departmental Performance Report of the Office of the Auditor General of Canada 
(Adopted by the Committee on 4 June 2009; presented to the House on 16 June 2009) 


Report 17—Chapter 1, National Security: Intelligence and Information Sharing, of the 2009 Status 
Report of the Auditor General of Canada (Adopted by the Committee on 16 June 2009; presented to 
the House on 19 June 2009) 


Report 18—Chapter 4, Managing Risks to Canada's Plant Resources—Canadian Food Inspection Agency 
of the December 2008 Report of the Auditor General of Canada (Adopted by the Committee on 
16 June 2009; presented to the House on 19 June 2009) 
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Appendix C Report on the audit of the President of the Treasury Board’s report /ab/ing of Crown 
Corporations Reports in Parliament | 


Tablings in Parliament for parent Crown corporations: Annual reports and summaries of corporate plans and budgets 


Section 152 of the Financial Administration Act (the Act) requires the President of the Treasury Board to 
lay before each House of Parliament a report on the timing of the tabling, by appropriate ministers, 

of annual reports and summaries of corporate plans and of budgets of Crown corporations. This report 
of the President of the Treasury Board is included in the 2009 Annual Report to Parliament—Crown 
Corporations and Other Corporate Interests of Canada, which must be tabled by 31 December. 


The Act requires the Auditor General to audit the accuracy of the President of the Treasury Board's report 
on the timing of tablings and to present the results in her annual report to the House of Commons. 


At the time that our annual report was going to print, we were unable to include the results of the above 
audit, since the President of the Treasury Board's report had not yet been finalized. The auditor's report, 
which is required by the Act, will therefore be included in the next Report of the Auditor General to the 
House of Commons. It will also be appended to this year's report of the President of the Treasury Board. 
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Appendix D Costs of Crown corporation audits conducted by the Office of the Auditor General of Canada 


The Office is required, under section 147 of the Financial Administration Act, to disclose its costs incurred 
in preparing annual audit (Exhibit D.1) and special examination reports on Crown corporations. 


An audit report includes an opinion on a corporation's financial statements and on its compliance 
with specified authorities. It may also include reporting on any other matter deemed significant. 


A special examination determines whether a corporation's financial and management control and 
information systems and its management practices provide reasonable assurance that 


* assets have been safeguarded and controlled; 
¢ financial, human, and physical resources have been managed economically and efficiently; and 
* operations have been carried out effectively. 


In 2008-09, the Office completed the special examination of eight Crown corporations. The costs 
incurred are in the following table: 


Canada Council for the Arts $757,009 
Canada Development Investment Corporation (joint auditor) $328,107 
Canada Mortgage and Housing Corporation (joint auditor) $807,924 
Defence Construction (1951) Limited $461,995 
Federal Bridge Corporation Limited , The $798,768 
Pacific Pilotage Authority $214,687 
Parc Downsview Park Inc. $459 381 
VIA Rail Canada Inc. $1,247,055 
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Exhibit D.1 Cost of preparing annual audit reports for fiscal years ending on or before 31 March 2009 


yearended | 


ete: 


lotage Authority 31.12.08 | $94,327 
Atomic Energy of Canada Limited (joint auditor) 31,0309 ina 392/155 
Blue Water Bridge Authority : 31.08.08 161,242 
Business Development Bank of Canada (joint auditor) Sle OSeO9 392,235 
Canada Council for the Arts 31:03:09 1815059 
Canada Deposit Insurance Corporation a 31.03.09 138,946 
Canada Development Investment Corporation (joint auditor) 31.12.08 103,166 
Canada Hibernia Holding Corporation SL WZ OS 95,970 
Canada Lands Company Limited (joint auditor) 31.03.09 308,150 
Canada Mortgage and Housing Corporation (joint auditor) | SlmZ20S 676,241 
Canada Post Corporation (joint auditor) Sle 08 709,329 
Canadian Air Transport Security Authority 31.03.09 | 502,045 
Canadian Broadcasting Corporation 31.03.09 839,974 
- Canadian Commercial Corporation Sil OS OS 217,085 
Canadian Dairy Commission j Sul LO/ Oks} Zoe STS 
Canadian Museum for Human Rights — - Si03709 141,345 
Canadian Museum of Civilization 31.03.09 184,635 
Canadian Museum of Nature 31,03;09 132,091 
Canadian Race Relations Foundation 31.03.09 136,945 
Canadian Tourism Commission S1,12:08 312,802 
Cape Breton Development Corporation . 31.03.09 108,001 
Defence Construction (1951) Limited 3103.09 | 97 Je 
Enterprise Cape Breton Corporation SRO S.09 138,277 
Export Development Canada a | 31.12.08 137,411 
Farm Credit Canada 31.03.09 744,456 
Federal Bridge Corporation Limited, The SieOSnOg 97,879 
First Nations Statistical Institute | 31:03:09 OS is flo 
Freshwater Fish Marketing Corporation 30.04.08 | 264,494 
Great Lakes Pilotage Authority 31.12.08 127,700 
International Development Research Centre | 31.03.09 190,483 
Jacques Cartier and Champlain Bridges Incorporated, The 31.03.09 110,153 ; 
Laurentian Pilotage Authority 31712,08 116,528 
Marine Atlantic Inc. 31.03.09 | . 203,960 
National Arts Centre Corporation 31.08.08 =| 312,585 
National Capital Commission 31.03.09 269,337 
National Gallery of Canada 31.03.09 | 158,893, 
National Museum of Science and Technology : 31.03.09 Pad (128,189 — 
Old Port of Montréal Corporation Inc. 31.03.09, | 164,819 
Pacific Pilotage Authority | «31.12.08 | 74,846 
Parc Downsview Park Inc. 31.03.09 156,555 _ 
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Exhibit D.1 Cost of preparing annual audit reports for fiscal years ending on or before 31 March 2009 (continued) 


Bee ore) 


Investment Board (joint auditor) = = © ° PESOS 09 


Public Sector Pension 526,492 
Public-Private Partnerships Canada Inc. (joint auditor) coe 31.03.09 85,856 
Ridley Terminals Inc. : 31512,08 207,687 
Royal Canadian Mint ; Sh ll Ole: 662,909 
Seaway International Bridge Corporation Ltd., The 31.03.09 83,855 
Standards Council of Canada Bl OStO9 73,749 
~ Telefilm Canada 31.03.09 180,231 
VIA Rail Canada Inc. SLaIZA08 67/65423 


* Audit costs are preliminary and subject to year-end adjustments 
** New audit client, audit not yet complete. 
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What we examined 


Why it’s important 


Evaluating the Effectiveness 
of Programs 


Main Points” 


Effectiveness evaluation is an established tool that uses systematic 
research methods drawn from many different disciplines to assess how 
well a program is achieving its objectives. When done well, it allows 
departments to develop evidence to determine how well their 
programs meet expectations, and whether they are cost-effective. Over 
the past 40 years, the federal government has made efforts to embed 
the management practice of evaluating program effectiveness, as an 
essential part of its support for program evaluation. 


The 2006 Federal Accountability Act enacted into law a requirement 
that all grant and contribution programs be evaluated every five years. 
The new Policy on Evaluation that came into effect in 2009 extends 
the requirement for evaluation to cover all direct program spending 
over a five-year cycle. 


We examined how evaluation units in six departments identify and 
respond to the various needs for effectiveness evaluations. We also 
looked at whether they have built the required capacity to respond to 
those needs. In addition, we looked at the oversight and support role of 
the Treasury Board of Canada Secretariat in monitoring and improving 
the evaluation function in the government, specifically with respect to 
effectiveness evaluations. The period covered by our audit was 2004 


to 2009. 


Governments are under continual pressure to spend money on a range 
of programs designed to serve particular needs of society. While many 
factors affect the decisions that governments must ultimately make 
about programs, effectiveness evaluations can aid their decision 
making by providing objective and reliable information that helps 
identify programs that are working as intended; those that are no 
longer needed; and those that are not accomplishing the desired 
objectives and could be replaced by programs that will achieve the 
objectives more cost-effectively. In addition, effectiveness evaluation is 
expected to serve the information needs of parliamentarians. 


Report of the Auditor General of Canada—Fall 2009 


Chapter 1 ee rah 


EVALUATING THE EFFECTIVENESS OF PROGRAMS 


2 Chapter 1 


What we found 


One of the most important benefits of effectiveness evaluation is to 
help departments and agencies improve the extent to which their 
programs achieve their objectives. Departments need to demonstrate 
to Parliament and taxpayers that they are delivering results for 
Canadians with the money entrusted to them. 


- The six departments we examined followed systematic processes to 


plan their effectiveness evaluations and completed most of the 
evaluations they had planned. However, over the audited period, 
each department’s evaluations covered a relatively low proportion of 
its total program expenses—between five and thirteen percent 
annually across the six departments. 


- In effect, the rate of coverage was even lower because many of the . 


effectiveness evaluations we reviewed did not adequately assess 
program effectiveness. Often departments have not gathered the 
performance information needed to evaluate whether programs are 
effective. Of the 23 evaluation reports we reviewed, 17 noted that 
the analysis was hampered by inadequate data, limiting the 
assessment of program effectiveness. 


- The departments we examined told us that it remains a challenge to 


find experienced evaluators, and they have made extensive use of 
contractors to meet requirements. Departments expressed concern 
about their capacity to start in 2013 to evaluate all direct program 
spending, as required by the 2009 Policy on Evaluation. To ensure 
full coverage (which includes grants and contributions), they will 
have to evaluate an average of 20 percent of their direct program 
spending each year of the five-year cycle. 


¢ The Treasury Board of Canada Secretariat introduced initiatives 


over the past few years to address the need for improvements in 
evaluation across the government. However, it did not provide 
sustained support for effectiveness evaluation. In particular, it made 
little progress on developing tools to assist departments with the 
long-standing problem of a lack of sufficient data for evaluating 
program effectiveness. 


- With the exception of Environment Canada, which has processes in 


place to identify needed improvements, the audited departments do 
not regularly identify and address weaknesses in effectiveness 
evaluation. 


The Secretariat and the departments have responded. The Treasury 
Board of Canada Secretariat and the departments agree with our 
recommendations. Their detailed responses follow each 
recommendation throughout the chapter. 
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Introduction 


1.1 The 2009 Treasury Board Policy on Evaluation defines 
evaluation as “the systematic collection and analysis of evidence on 
the outcomes of programs to make judgments about their relevance, 
performance and alternative ways to deliver programs or to achieve the 
same results.” The objective of this policy is to create a comprehensive 
and reliable base of evaluation evidence that is used to support policy 
and program improvement, expenditure management, Cabinet 
decision making, and public reporting. 


1.2 Program evaluation is recognized as a key source of information on 
program effectiveness. This information is essential if senior officials are 
to base program and funding decisions on evidence of program 
effectiveness. Potential key users of evaluation findings include senior 
department officials, the managers of programs being evaluated, central 
agencies, and parliamentarians. The findings of program evaluation may 
also be of interest to program stakeholders and to the public. 


13 Program evaluation has been practised in the federal 
government, in one form or another, for close to 40 years. The Office 
of the Auditor General examined program evaluation in 1978, 1983, 
1986, 1993, 1996, and 2000. The history of federal program evaluation 
from 1970 to 2000 reveals repeated initiatives at the centre of 
government to establish and support the function, and often critical 
observations by the Office of the Auditor General on the success of 


these efforts (Exhibit 1.1). 


14 The advent of results-based management. In 2000, the 
Treasury Board of Canada Secretariat issued Results for Canadians: 

A Management Framework for the Government of Canada. It outlined a 
results-based management approach, which states that 


departments and agencies need to implement an information 
regime that measures, evaluates and reports on key aspects of 
programs and their performance in core areas; holds managers 
accountable for achieving results; and ensures unbiased analysis, 
showing both good and bad performance. 


15 Aswe noted in our report from the same year (the December 2000 
Report of the Auditor General of Canada, Chapter 20—Managing 
Departments for Results and Managing Horizontal Issues for Results), 
evaluation has an important role to play in managing for results. It can 
provide important information on program performance that is not 
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gathered by ongoing monitoring systems, and can help managers 
understand why programs are working or not. 


1.6 In 2001, the Treasury Board issued an evaluation policy, 
intended to reflect the results-based management philosophy 
expressed in Results for Canadians. The objective of this policy was to 
ensure that the government has information on the performance of its 
policies, programs, and initiatives that is timely, strategically focused, 
objective, and evidence-based. The Policy described evaluation as a 


Exhibit 1.1 The history of federal program evaluation reveals many audit observations that are critical 
of government initiatives 
Year Event 


1970 The Treasury Board of Canada Secretariat established a Planning 
Branch to carry out interdepartmental studies of policy programs. 


LOW): The Treasury Board issued the first evaluation policy, which required 
departments to subject each program to effectiveness evaluation on a 
regular basis. 


1978 The Auditor General’s Study of Procedures in Cost-Effectiveness 
reviewed 23 programs in 18 departments, and found few successful 
attempts to evaluate their effectiveness. 


1981 The Office of the Comptroller General developed a policy framework to 
guide and structure departmental evaluation functions, and established 
a team of liaison officers to give guidance and advice to departments in 
its implementation. 


1983 The Auditor General’s report found that although very real progress had 
been made in developing the program evaluation function since its 
1978 study, few high-quality evaluations were being done. 


1986 The Program Review carried out by the Nielsen Task Force found that. 
the information received from program evaluations was “generally 
useless and inadequate.” 


1986 The Auditor General’s follow-up to the 1983 audit concluded that there 
had been an improvement in the quality of methodology and: reporting 
of program evaluation, but that problems regarding balance and full 
disclosure of limitations in methods still existed. 


1993 The Auditor General found that “high expectations and great potential” 
had been only partly fulfilled: “The system’s results are often 
disappointing. Program evaluations frequently were not timely or 
relevant. Many large-expenditure programs had not been evaluated 
under the policy...” 


1996 -The Auditor General’s follow-up to the 1993 audit found that little 
progress had been made to address effectiveness issues. 


2000 The Auditor General found that the evaluation function had actually 
regressed, due in part to reductions in funding that undermined 


capacity. 
ee hn i a a 2 Se ee a eg 
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Direct program spending—lIncludes operating 
and capital spending and grants and 
contributions, but does not include public debt 
charges and major transfers to persons or other 
levels of government. 


Reference level—The amount of funding that 
the Treasury Board has approved for 
departments and agencies to carry out approved 
policies and programs for each year of the 
planning period. 
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management tool for the periodic assessment of a program’s 
effectiveness in achieving objectives, of its impact and relevance, and 
of alternative ways to achieve expected results (cost effectiveness). 


17 In 2003, the Treasury Board of Canada Secretariat introduced 
the Management Accountability Framework, to assess management 
performance in departments, including program evaluation. 
Subsequent guidance that was issued in 2007 to support the 2005 
Policy on Management, Resources and Results Structure (MRRS), 
encouraged department heads of evaluation to provide advice on 
the Performance Measurement Framework embedded in their 
organization’s MRRS. The Secretariat views the Framework and 
the policy requirements as an integrated approach to measuring 
management performance in departments. 


18 Focus on expenditure management. More recent developments 
have placed renewed emphasis on the expenditure management role 
of program evaluation. As described in its 2007 and 2008 Budget 
documents, the federal government has introduced a new expenditure 
management system, a key pillar of which is the ongoing assessment 
of all direct program spending, known as strategic reviews. Strategic 
reviews are intended to ensure that programs are effective and 
efficient, meet the priorities of Canadians, and are aligned with core 
federal responsibilities. According to the Treasury Board of Canada 
Secretariat, program evaluation is a key source of information on 
program effectiveness in support of these reviews. 


19 The 2009 Policy on Evaluation. The most recent development 
in federal program evaluation is a new policy that was approved in 
April 2009. In addition to supporting the renewal of the Expenditure 
Management System, by improving the information base for strategic 
reviews of departmental spending, the new policy requires that 
evaluations cover all direct program spending over a five-year cycle. 
Each year, departments will have to evaluate an average of 20 percent 
of their direct program spending to ensure full evaluation coverage, 
which includes grants and contributions programs. Larger departments 
and agencies must implement this requirement from | April 2013. 
Following a four-year transition period, the first cycle covers 

the 2013-14 to 2017-18 fiscal years. Smaller organizations, with fewer 
than 500 full-time equivalents and a reference level of less than 

$300 million, are subject to the lesser requirement of ensuring 
coverage as appropriate. Accompanying the new policy is a directive 
on evaluation that defines the roles and responsibilities of department 
officials involved in evaluation, and standards that set minimum 
requirements for the quality, neutrality, and usefulness of evaluations. 


Chapter 1 | x) 


EVALUATING THE EFFECTIVENESS OF PROGRAMS 


Effectiveness evaluation—An assessment 
of the extent to which programs are relevant, 
produce impacts and outcomes, achieve their 
objectives, and are cost-effective. Other types 
of evaluation examine program implementation 
and management. 


Meeting needs for effectiveness 
evaluation 


Focus of the audit 


1.10 This was a government-wide audit that examined program 
evaluation in relation to the measurement of program 
effectiveness—effectiveness evaluation. The overall objective of this 
audit was to determine whether selected departments and the Treasury 
Board of Canada Secretariat are meeting the needs for effectiveness 
evaluation and are identifying and making improvements in 
effectiveness evaluation. 


111 The audit covered a five-year period, from the 2004—05 

to 2008-09 fiscal years. Focusing on this period allowed us to examine 
the impact of Secretariat efforts, through a series of its studies, to 
understand how well the 2001 evaluation policy was working up to 
the 2004—05 fiscal year. We also examined selected departments’ 
efforts to address any problems identified over the same period. 
Because the 2001 evaluation policy was replaced in April 2009, 

the audit did not focus directly on compliance with either policy. 


112 Detailed examination work was carried out in the Treasury 
Board of Canada Secretariat and in six departments: 


¢ Agriculture and Agri-food Canada, 

¢ Canadian Heritage, 

¢ Citizenship and Immigration Canada, 
¢ Environment Canada, 

¢ Fisheries and Oceans Canada, and 


¢ Human Resources and Skills Development Canada. 


1.13 More details on the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


1.14 Well-managed organizations operate according to a management 
cycle for continuous improvement consisting of planning, doing, 
checking, and improving. We looked for evidence that the audited 
departments planned their effectiveness evaluations to meet identified 
needs, executed these plans, checked to see that needs were met, and 
made improvements where required. We followed a similar approach in 
relation to the Treasury Board of Canada Secretariat’s oversight and 
support role (paragraphs 1.63—1.93). 
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115 We found that, in light of their limited evaluation coverage and 
reliance on insufficient performance information, departments were 
not able to demonstrate that they are fully meeting needs for 
effectiveness evaluation. 


The need for effectiveness evaluations continues to grow 


1.16 When departments plan their effectiveness evaluation work, 
they must first identify the needs for these evaluations. Heads of 
evaluation in the six departments told us that they based those needs 
on both government-wide requirements and internal corporate risk- 
based needs. 


1.17 The government views evaluation as the primary source of 
neutral and systematic information on the ongoing relevance and 
performance of policies and programs. It also expects evaluation to 
show alternative ways of achieving expected results and program design 
improvements. Government-wide requirements for effectiveness 
evaluations are defined both in statute and in policy (Exhibit 1.2). 


118 According to the Treasury Board of Canada Secretariat, a critical 
need for effectiveness evaluation arises from the strategic review 
process in the Expenditure Management System. Organizations are 
required to conduct strategic reviews of direct program spending every 


Exhibit 1.2 Government-wide requirements for effectiveness evaluation appear in both statutes 
and policies 


The 2000 Policy on Transfer Payments (revised in 2008) required departments to 
review and report on the effectiveness of transfer payments when requesting renewal of 
program terms and conditions. The Financial Administration Act (amended in 2006) 
requires departments to conduct a review every five years of the relevance and 
effectiveness of each ongoing program of grants or contributions for which they are 
responsible. 


The 2001 Treasury Board Evaluation Policy advised that evaluations should consider 
the relevance, success, and cost-effectiveness of programs. 


Approved program funding may be subject to specific conditions in Treasury Board 
submissions and memoranda to Cabinet requiring the department to evaluate the 
program and report back to the Treasury Board or Cabinet before a set deadline. 


In the 2007 Budget, the Government announced its new Expenditure Management 
System (EMS), including enhanced requirements for evaluation. 


As of April 2007, evaluation of effectiveness is required for selected new federal 
regulations. 


The 2009 Treasury Board Policy on Evaluation requires that all evaluations that are 
intended to count toward coverage requirements address value for money by including 
clear and valid conclusions about the relevance and performance of programs. 
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four years to ensure that the government is directing its resources to 
the highest priority requirements. 


1.19 In addition to these government-wide requirements, heads of 
evaluation identified internal corporate risk-based needs for 
effectiveness evaluations that respond to departmental priorities and 
to the priorities of government. These priorities are typically defined 
through consultation with senior departmental management. 


1.20 Between 2004 and 2009, the need for effectiveness evaluations 
had grown. This was largely due to measures the government brought 
forward under the Federal Accountability Act in 2006. This Act 
introduced an amendment to the Financial Administration Act requiring 
that, every five years, all ongoing non-statutory grant and contribution 
programs be evaluated or reviewed for relevance and effectiveness. 


Departments have systematic processes to plan effectiveness evaluation 


1.21 In light of the many needs for effectiveness evaluation and 
their potential importance in informing decision making, it is critical 
that departments consider these needs when making program 
evaluation plans. 


1.22 We expected program evaluation plans in the six departments to 
consider both government-wide requirements and corporate risk-based 
needs for effectiveness evaluation. We found that all departments had 
developed risk-based plans during the audited period (some started in 
the 2004-05 fiscal year and others started in 2005-06). Most of these 
plans are produced annually, and most of the recent ones cover a 
five-year period. We found evidence that, to identify corporate 
priorities, each department consulted its senior management and its 
various program areas when developing its evaluation plans. We also 
found that all departments—except Agriculture and Agri-Food 
Canada and Citizenship and Immigration Canada—had processes in 
place to identify and track requirements for grant and contribution 
renewals in Treasury Board submissions. 


Expenditure management needs were not adequately considered 


1.23 Officials in the six departments told us that the need for strategic 
review in the Expenditure Management System had not been a key 
consideration during previous years’ evaluation planning. However, 
they told us that this need is now being considered when preparing 
their evaluation plans. To date, of the six departments audited, only 
Canadian Heritage (in 2007) and Agriculture and Agri-Food Canada 
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(in 2008) have completed a strategic review. Officials at Fisheries and 
Oceans Canada and at Canadian Heritage informed us that they plan 
to undertake evaluations in preparation for the next strategic review. 


Departments complete most planned evaluations 


1.24 ‘To meet the needs for effectiveness evaluations, we expected the 
six departments to have conducted the evaluations that they had 
identified in their plans. We found that most of the evaluations 
planned by the six departments were carried out (Exhibit 1.3). 
Department officials told us that some evaluations were not completed 
for a variety of reasons, including program cancellations and redesign, 
limited data availability, and changes in internal client needs (for 
example, senior management and program managers) and evaluation 
capacity. Officials at Human Resources and Skills Development 
Canada informed us that, because many of the evaluations not 
completed during the audit were multi-year evaluations with planned 
completion dates beyond our audit period, they projected a higher 
completion rate. 


Exhibit 1.3 Most of the planned evaluations were completed 


se Numberob (2s 

; | planned =. | Number and percentage: 

Department evaluations* | completed** — 
Canadian Heritage 48 42 (88%) 
Fisheries and Oceans Canada 30 24 (80%) 
Environment Canada 29 23 (79%) 
Citizenship and Immigration Canada 21 1519s) 

+ = 

Agriculture and Agri-Food Canada 25 14 (56%) 
Human Resources and Skills Development 64 34 (53%) 
Canada 


* Based on departmental evaluation plans between the 2004-05 and 2007-08 fiscal years. 


** Based on listings of evaluations provided by departments in their 2004-05 to 2007-08 plans, which 
were completed by 30 April 2009. 


Evaluation coverage of programs is limited 


1.25 While the 2009 Policy on Evaluation requires departments to 
evaluate all direct program spending, there was no such requirement 
during the period of our audit. The Treasury Board of Canada 
Secretariat required departments to conduct evaluations based on risk. 
We noted that the Secretariat encouraged departments to embed 


Report of the Auditor General of Canada—Fall 2009 


Chapter 1 | 3g 


EVALUATING THE EFFECTIVENESS OF PROGRAMS 


Non-statutory grant and contribution 
programs—Programis whose spending 
authority is provided in an appropriation act that 
is voted on in Parliament in the Main and 
Supplementary Estimates, as opposed to 
programs whose spending authority comes from 
other legislation. 


program evaluations into program management and ensure adequate 
evaluation coverage of programs. The six departments followed 
systematic processes when they planned evaluations, including 
effectiveness evaluations. However, within the departments we found 
that a low proportion of total program expenses were evaluated. 


1.26 We examined the data that the six departments provided to the 
Treasury Board of Canada Secretariat as part of the Annual Capacity 
Assessment Survey, including information on the annual expenses of 
evaluated programs. We compared these program expenses to total 
departmental program expenses for that year. We found wide variation 
across the six depattments in their evaluation coverage of their 
program expenses (Exhibit 1.4). 


Exhibit 1.4 A low proportion of total program expenses were evaluated between the 2004—05 and 
2007-08 fiscal years 


Estimated average annual 
percentage of program 
Department | expenses evaluated 
Canadian Heritage : | 13% 
Agriculture and Agri-Food Canada* 11% 
Environment Canada 9% 
Fisheries and Oceans Canada 8% 
Citizenship and Immigration Canada et ae 
Human Resources and Skills Development Canada* 5% 


* Agriculture and Agri-Food Canada and Human Resources and Skills Development Canada each carried . 
out evaluation work that included several programs. This has increased their coverage. 


Sources: Treasury Board of Canada Secretariat Annual Capacity Assessment Survey, descriptive 
information provided by departments, and Public Accounts of Canada 


1.27 Our calculations showed that Canadian Heritage had the 
highest percentage of program expenses that were evaluated compared 
with the other audited departments. Notably Canadian Heritage also 
has the highest percentage of expenses on grant and contribution 
programs, which amounted to 80 percent of its total expenses in 

the 2007-08 fiscal year. According to the requirement under the 
Financial Administration Act, departments have until the 2011-12 fiscal 
year to evaluate all of their ongoing, non-statutory grant and 
contribution programs that were in existence in December 2006. 


1.28 As well, the requirements for renewals of grants and contributions 
occur in a concentrated period. For example, officials at Citizenship and 
Immigration Canada told us that it is challenging for them to meet their 
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evaluation requirements because the majority of grant and contribution 
programs come up for renewal at the same time, creating a spike in the 
evaluation unit’s workload. However, the Department indicated that 
these requirements were met over the period of our audit. We noted 
that, in the 2009-10 fiscal year, its evaluation unit plans to evaluate 
about 90 percent of its grant and contribution expenses, leaving little 
capacity to respond to other needs. 


Evaluations do not adequately assess effectiveness 


1.29 The quality of evaluation methods is a long-standing concern in 
the federal government. In 2005, the Treasury Board of Canada 
Secretariat reviewed the quality of evaluation reports to determine 
whether they had improved. The review noted that, while evaluation 
reports had improved in quality since 2002, there remained a “pressing 
need for further improvement.” 


1.30 Evaluation practitioners employ a range of methods. There are 
no universally accepted minimum standards for effectiveness 
evaluation. However, both the 2001 and 2009 Treasury Board policies 
on evaluation refer to the need to collect reliable data to support 
evaluation findings. The 2001 policy refers to objective data collection 
and analysis, while the 2009 Standard on Evaluation requires that 
evaluations be based on “multiple lines of evidence, including evidence 
produced from both quantitative and qualitative analysis.” 


1.31 Both policies recognize that the level of methodological rigour 
should reflect the intended use of the findings. In addition, in programs 
with low materiality, less rigorous methods may be appropriate. 


1.32 In 2005, the Treasury Board of Canada Secretariat found that 
evaluations tended to rely mainly on information from stakeholder 
interviews, file and document reviews, case studies, and surveys. 

The Secretariat also found that information from ongoing program 
performance measurement tended to be unavailable or insufficient to 
effectively support evaluations. According to the Secretariat, this was 
because few programs had reliable systems for collecting and reporting 
this information. The Secretariat’s review concluded that the lack of 
program performance data reduces the overall quality of evaluation 


reports. 


1.33 When data is not available, evaluators may have to collect it 
themselves, or find alternative sources for the information, and their 
ability to apply the appropriate methods may be constrained. Using a 
variety of methods, both quantitative and qualitative, is important 
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to ensure that evaluation methods generate enough reliable evidence. 
Unreliable data reduces confidence in the usefulness of evaluation 
measurements. 


1.34. We reviewed a sample of 23 evaluations identified by the 

six departments as addressing effectiveness to determine the types 

of data collected to support evaluation findings. We found that, of 

the 23 evaluations, 17 explicitly stated that program performance 
information was lacking because data was unavailable or was not 
sufficiently reliable. As a result, in 9 of 17 cases, the evaluations 
indicated that they were limited in their assessment of program success 
and effectiveness. Furthermore, in 6 of the 17 cases, the assessment 
was primarily based on interviews with program staff and stakeholders 


(Exhibit 1.5). 


Exhibit 1.5 Some effectiveness evaluations have insufficient performance information 


Agriculture and Agri-Food Canada. The Department’s Prairie Grain Roads Program 
operated between April 2001 and March 2006 with an average annual budget of $35 
million. The program was meant to assist Prairie provinces to address increased 
pressure on rural roads by improving roads, increasing truck tonnage capacity, and also 
increasing safety for road users. 


Completed in 2006, this $103,600 evaluation was to determine the program’s results 
and impacts, adequacy of design, continued relevance, and cost-effectiveness. 
Achieving these objectives was hindered by the program’s lack of performance 
measures to determine its effectiveness. It was not possible to know, for example, 
whether road safety had improved as a result of the program. 


Claims of improved safety were primarily based on interviews and satisfaction ratings 
from successful program applicants. If performance measures had been implemented 
from the outset of the program, more complete conclusions on the effectiveness of this 
program could have been reached. 


Citizenship and Immigration Canada. The Department’s Private Sponsorship of 
Refugees Program, which began in 1978, is intended to assist refugees to settle and 
build new lives in Canada through sponsorship by Canadian citizens. The annual 
budget of the program is approximately $5 million. 


The most recent evaluation of this program was done in 2007 at a cost of $268,000. 
The evaluation was intended to examine the program’s continued relevance, success in 
achieving outcomes as identified in its results-based management accountability 
framework, and its cost-effectiveness. 


This evaluation used data originating from a wide range of sources, including statistical 
information from both the department and Statistics Canada. As a result, the 
evaluation was able to more objectively document the extent to which refugees were 
finding employment, accommodation, and essential services. 


Fisheries and Oceans Canada. In 2006, the Department completed an evaluation of 
its Program for Sustainable Aquaculture. The program was launched in 2000; it had 
an annual budget in the 2008-09 fiscal year of $15 million. The objective of this 
program is to foster growth of a sustainable and competitive aquaculture industry and 
to increase public confidence in aquaculture. 
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Exhibit 1.5 Some effectiveness evaluations have insufficient performance information (continued) 


The stated purpose of this $98,600 evaluation was to examine the relevance, success, 
and cost-effectiveness of the program. However, the evaluators acknowledged that 
there was no ongoing monitoring system to track program results. For example, 
because there was limited information on the impact of aquaculture on human health, 
it was not possible to determine whether related program objectives were achieved. 


Without this information, many of the evaluation’s conclusions were based on 
interviews with federal officials and industry representatives, as well as on document 
and file reviews. As a result, this evaluation did not adequately address the 
effectiveness of the program. 


1.35 The heads of evaluation in all six departments confirmed that 
performance information they needed to evaluate whether programs 
are cost-effective and are achieving expected results was often 
insufficient. The development and implementation of ongoing 
performance measures is the responsibility of program managers, 
not departmental evaluation units. 


1.36 Due to the weaknesses in performance information and the need 
to apply appropriate evaluation methods, the actual coverage of 
departmental programs by effectiveness evaluations is even more 
limited than shown in this audit (Exhibit 1.4). Based on our sample, 
three quarters of the evaluations were hampered in their assessment of 
program effectiveness because of inadequate data. 


1.37 Recommendation. Agriculture and Agri-food Canada, 
Canadian Heritage, Citizenship and Immigration Canada, 
Environment Canada, Fisheries and Oceans Canada, and Human 
Resources and Skills Development Canada should develop and 
implement action plans to ensure that ongoing program performance 
information is collected to support effectiveness evaluation. 


Agriculture and Agri-Food Canada’s response. Agreed. 

The Department agrees that the systematic collection of program 
performance data by managers is necessary to report on program 
performance and to support effectiveness evaluations. 


As required by the Treasury Board Policy on Transfer Payments, 

a performance measurement strategy for ongoing management of 
transfer payment programs, including performance measures and 
indicators and a data collection strategy, is developed for each new 


transfer payment program. 
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The Department’s evaluation function reviews program performance 
measurement strategies as they are developed to ensure that outcomes 
are defined, measurable, and attributable and that the strategies, 

if implemented, are sufficient to support future evaluation work. 


Beginning in the 2009-10 fiscal year, the Department will conduct 
annual state of performance measurement reviews. The first such 
review will assess performance measurement practices at the 
Department and the adequacy of data collected for programs soon to 
be evaluated. An action plan to address its recommendations will be 
developed and its implementation monitored with a view to 
strengthening the Department’s practices in this area. 


Canadian Heritage’s response. Agreed. The Department is executing 
its Action Plan for Implementation of the Management, Resources and 
Results Structures (MRRS) Policy to ensure that program staff are able 
to fulfill their responsibility for developing and maintaining 
performance measurement strategies. Action plan measures include 


¢ the provision of information sessions and workshops, 
¢ the establishment of indicators and corresponding targets, 


¢ the development of robust methodologies to demonstrate 
outcomes, 


¢ the establishment of a centre of expertise on performance 
measurement within the Department, 


* the design and implementation of adequate tools and guidelines, 


* the establishment of relevant information technology and systems, 
and 


¢ the regular analysis and reporting of collected data. 


This action plan is expected to be completed by end of the 2011-12 
fiscal year. 


The Department’s Office of the Chief Audit and Evaluation Executive 
is continually providing advice and support to Department managers » 
in their efforts to implement this action plan. In line with the Transfer 
Payment Policy, this office is also providing timely advice and support 
on program design and performance measurement strategies through 
the review of official approval documents for the creation and renewal 
of new or existing programs. Finally, as required under the new 
Evaluation Policy, the office will be submitting, in the 2010-11 fiscal 
year, its first annual report to the Departmental Evaluation Committee 
on the state of performance measurement in the Department. 
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Citizenship and Immigration Canada’s response. Agreed. The 
Department recognizes the value of ongoing performance information 
for evaluation and will continue to support related departmental 
activities. The Department will develop an action plan that includes the 
renewal of the comprehensive departmental performance measurement 
framework and program activity architecture, and furthers the 
integration of the Framework into the business planning process. 


Environment Canada’s response. Agreed. The Department accepts 
this recommendation, which echoes the intent of the 2009 Evaluation 
Policy related to performance information. As such, actions are already 
under way within Environment Canada to implement this 
recommendation. These include ongoing monitoring of the 
implementation of management responses to previous evaluations that 
have identified concerns with performance information, and the 
development and implementation of a strategy to inform all 
department managers of the Evaluation Policy requirements pertaining 
to performance measurement. 


In addition, the Department’s evaluation plan will be expanded to 
include a monitoring component to verify, within available resources, 
the status of performance data collected in the department and whether 
sufficient performance information will be available to support 
upcoming evaluations. This monitoring component will be included in 
the 2010-15 Evaluation Plan and will be updated annually thereafter. 


Further, for the 2010-11 fiscal year, the Department’s performance 


-measurement framework has been linked to the Department’s program 


activity architecture, in that performance measures have been 
identified for all programs. 


Fisheries and Oceans Canada’s response. Agreed. The Department's 
performance measurement framework links its core indicators to the 
departmental program activity architecture (PAA), thus identifying 
performance measures for all program activities and sub-activities. 
Each fiscal year, the Department conducts an analysis of the state of 
performance measurement in the Department and provides an annual 
report to the Departmental Evaluation Committee. In addition, the 
Department will develop and implement an action plan to ensure that 
ongoing program performance information is collected to support 
effectiveness evaluation by the end of August 2010. | 


Human Resources and Skills Development Canada’s response. 
Agreed. The Department accepts this recommendation that echoes 
the intent of various Treasury Board of Canada Secretariat policies, 
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including the 2009 Evaluation Policy. As such, the Department already 
gathers and monitors ongoing performance information to support 
effectiveness evaluation, including 


* monitoring implementation of management responses to previous 
evaluations that have identified data issues for current programs; 


° undertaking early evaluative work in advance of the formal 
initiation of effectiveness evaluations as part of the evaluation 
planning process, to review the state of performance data and 
logic models; and 


* monitoring the implementation of new programs (for example, 
Economic Action Plan initiatives) to ensure the necessary 
administrative and performance data are available to support 
future evaluation activities. 


The Department is also undertaking a comprehensive review, 
refinement, and validation of its Performance Measurement 
Framework (PMF) to make it more robust and comprehensive, to 
support ongoing planning, monitoring, and managing for results and 
the Evaluation Directorate has been actively involved in this work. 
The progress made on the departmental PMF will support both 
performance monitoring as well as evaluation of relevance and 
effectiveness. 


Some quality assurance processes are in place 


1.38 Quality assurance processes are designed to ensure that quality 
requirements are being met. In the case of effectiveness evaluation, 
quality assurance helps to ensure that reports meet defined standards 
and provide decision makers with reliable and useful evaluation findings. 


1.39 We expected the six departments to demonstrate that they had 
developed systematic quality assurance processes for their effectiveness 
evaluations. We found that all six departments had quality assurance 
processes in place. While these processes were not identical in design, 
we identified a number of common elements: internal evaluation 
standards, internal review, and external expert review. All 

six departments did internal reviews and all but two (Agriculture and 
Agri-Food Canada and Fisheries and Oceans Canada) used external 
experts, either routinely or on a selective basis, to review for quality 
assurance. Despite these quality assurance processes, we found that 
three quarters of the evaluations in our sample failed to adequately 
address effectiveness. 


16 


Chapter 1 


Report of the Auditor General of Canada—Fall 2009 


EVALUATING THE EFFECTIVENESS OF PROGRAMS 


1.40 Quality assurance processes may also support the independence 
of the evaluation unit from program management. Through its 2001 
and 2009 evaluation policies, the Treasury Board acknowledged that 
having a departmental evaluation committee approve evaluation plans 
and reports supports independence, because these plans and reports 
are reviewed objectively by department officials who are not program 
managers. 


1.41 We examined the processes that the six departments followed to 
review and approve evaluation plans and reports. In all 

six departments, both plans and reports are approved by senior 
evaluation committees. 


142 We note that, unlike evaluation committees, the departments’ 
audit committees are required to have external members. In our view, 
this is a good practice, because the knowledge and perspectives of 
practitioners from outside government are being considered. This 
practice could also have merit for evaluation committees. Under 

the 2009 Policy on Evaluation, evaluation committees are required to 
review evaluation plans and reports and recommend their approval 

by the deputy head. Committees with external members could play a 
stronger role in the continuous improvement of effectiveness evaluation. 


1.43 Recommendation. Agriculture and Agri-food Canada, 
Canadian Heritage, Citizenship and Immigration Canada, 
Environment Canada, Fisheries and Oceans Canada, and Human 
Resources and Skills Development Canada should consider the merits 
of including external experts on their departmental evaluation 
committees. The Treasury Board of Canada Secretariat should provide 
guidance to departments in this regard. 


Agriculture and Agri-Food Canada’s response. Agreed. 

The Department has recently introduced a number of practices to 
ensure production of strong evaluation reports. They include seeking 
input from external experts on evaluations in progress. The Department 
will also consider including external members on its Departmental 
Evaluation Committee, in the context of guidance provided by 
Treasury Board of Canada Secretariat. 


Canadian Heritage’s response. Agreed. At the Department, oversight 
of the evaluation function is provided by the Strategic Policy, Planning 
and Evaluation Committee (SPPEC), chaired by the Deputy Head. 
This structure provides opportunities for enhanced integration 
between the policy, planning, and evaluation functions of the 
Department. As well, the Department already brings key evaluation 
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reports when necessary to its department audit committee for review 
and discussion. The Department will collaborate with the Treasury 
Board of Canada Secretariat’s Centre of Excellence in Evaluation to 
assess the value added of integrating the advice of external evaluation 
experts to inform the work of departmental evaluation committees. 


Citizenship and Immigration Canada’s response. Agreed. 
The Department will assess the benefits of including an external 
evaluation expert on the departmental evaluation committee. 


Environment Canada’s response. Agreed. The Department accepts 
the recommendation and will await guidance from the Treasury Board 
of Canada Secretariat on the inclusion of external members on 
departmental evaluation committees. 


Fisheries and Oceans Canada’s response. Agreed. The Department 
will consider the merit of including external members on its 
department evaluation committee, in the context of guidance provided 
by the Treasury Board of Canada Secretariat. 


Human Resources and Skills Development Canada’s response. 
Agreed. The Department has, in the past, included members from 
outside the Department on its departmental evaluation committee. 
The Department will reconsider the formal inclusion of external 
experts on the current Departmental Evaluation Committee and will 
look to the Treasury Board of Canada Secretariat for guidance on this 
part of the Evaluation Policy. 


The Treasury Board of Canada Secretariat’s response. Agreed. 

The Secretariat agrees that it should provide guidance to departments 
on the possible merits of including external experts on their 
departmental evaluation committees. These actions will be 


completed by 31 March 2010. 


Departments are not systematically identifying priorities for improvement 


144 The Treasury Board of Canada Secretariat carries out annual 
assessments of department management using indicators that measure 
each of the 10 elements of the Management Accountability 
Framework. Among these elements are assessments of evaluation 
coverage, quality, and use. According to the Secretariat, these 
assessments help deputy heads identify priorities for management 
improvement. In addition to these assessments, we expected the 

six departments to have their own internal processes for determining 
whether they are meeting needs for effectiveness evaluations. 
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1.45 As noted earlier (paragraph 1.22), we found that all departments 
have consultation processes aimed at ensuring that their evaluation 
plans reflect corporate priorities. In addition, we did see other 
improvements in some departments, although these were not 
systematic in nature. Only Environment Canada has a formal process 
in place to systematically identify aspects of its evaluation practice that 
require improvement. Environment Canada does a number of things to 
help ensure that its evaluation practice is oriented toward continuous 
improvement. For example, client feedback is solicited through post- 
evaluation surveys that provide ongoing feedback on the quality and 
value of evaluations. ; 


146 In addition, when Environment Canada’s evaluations are 
completed, lessons-learned exercises are often developed and shared 
with managers of similar programs and initiatives to enhance the 
overall utility of evaluation findings. The evaluation unit also 
developed a self-assessment framework for quality assurance that is 
based on a self-assessment guide for internal audit. The unit adapted 
some elements in the guide to apply them to evaluation and it also 
considered existing standards for evaluation. The results of these 
activities are communicated to the evaluation committee. 


1.47. With the exception of Environment Canada, the audited 
departments could not demonstrate that they have internal processes 
in place to systematically identify areas for improvement in 
effectiveness evaluation over the 2004—05 to 2008-09 period. Such a 
process enables departments to ensure that effectiveness evaluations 
are following the management cycle for continuous improvement and 
becoming more useful for making key decisions. 


1.48 Recommendation. Agriculture and Agri-food Canada, 
Canadian Heritage, Citizenship and Immigration Canada, Fisheries 
and Oceans Canada, and Human Resources and Skills Development 
Canada should implement systematic processes to determine whether 
their effectiveness evaluations are meeting government-wide 
requirements and internal corporate needs, and act on areas identified 
for improvement. The Treasury Board of Canada Secretariat should 
monitor and provide any additional support it considers necessary for 
the implementation of these processes. 


Agriculture and Agri-Food Canada’s response. Agreed. 

The Department accepts this recommendation and notes that in 
the past year, it has introduced a number of systematic processes, 
which together will ensure effectiveness evaluations address senior 
management information needs in a timely manner. They include 
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annual consultations on evaluation priorities, requests for feedback on 
completed evaluations, and annual Head of Evaluation reports on the 
performance of the evaluation function. 


Canadian Heritage’s response. Agreed. The Department recognizes 
the need to establish systematic processes to assess whether 
effectiveness evaluations are addressing needs. The Evaluation 
Services Directorate is already developing a performance measurement 
framework and management strategy to identify clear performance 
expectations and standards for the Department’s evaluation function. 
A systematic process to collect, analyze, and report on performance 
data and client satisfaction will be implemented in order to identify 
‘areas of improvements. Based on data collected during the first year 
of implernentation (the 2010-11 fiscal year), periodic reporting to the 
Departmental Evaluation Committee should begin by the 2011-12 
fiscal year. This performance information will complement data already 
reported in the context of the annual management accountability 
framework assessments conducted by the Treasury Board of 
Canada Secretariat. 


Citizenship and Immigration Canada’s response. Agreed. 

The Department will assess the need for systematic processes in 
addition to the Management Accountability Framework assessment 
process, the oversight of the Departmental Evaluation Committee, 
requirements for annual reporting, and the Department’s evaluation 
process, which includes several steps of consultation and feedback 
from the Department’s branches. | 


Fisheries and Oceans Canada’s response. Agreed. 

The recommendation echoes the intent of the 2009 Policy on 
Evaluation. As such, actions are already under way within the 
Department to implement a systematic process to determine whether 
effectiveness evaluations are meeting internal corporate needs and 
government-wide needs (i.e., a strategic review) and to act on areas 
identified for improvement. Further work in this area will be completed 
in the context of guidance provided by the Treasury Board of 

Canada Secretariat. 


Human Resources and Skills Development Canada’s response. 
Agreed. The Department currently employs a variety of systematic 
processes to ensure the quality and relevance of its evaluations for both 
internal and government-wide needs. External peer reviewers and 
evaluation advisory committees are a mandatory element of the 
evaluation work to ensure that evaluations are meeting information 
needs. Further work in systematically determining whether evaluations 
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are meeting government and senior management needs will be 
developed, building upon the current annual Management 
Accountability Framework assessment process led by the Treasury’ 
Board of Canada Secretariat. Areas for improvement will be identified 
and reported to the Departmental Evaluation Committee. 


The Treasury Board of Canada Secretariat’s response. Agreed. 

The Secretariat agrees that it should assist departments as necessary 
in their implementation of processes to determine whether evaluations 
are meeting government-wide needs and should provide support to 
departments that it considers necessary. 


The new Treasury Board Policy on Evaluation, which came into effect 
on | April 2009, includes specific requirements to enhance evaluation 
coverage and examination of program effectiveness as well as 
performance information to support evaluations. The policy calls on 

the Secretary of the Treasury Board to provide functional leadership for 
evaluation across government, including monitoring and reporting 
annually to the Treasury Board on the health of the evaluation function. 


The Secretariat currently carries out a large portion of this work 
through the annual Management Accountability Framework 
assessment process, which the Office of the Auditor General 
acknowledges was not covered by the current audit. The Secretariat 
communicates recommended areas for improvement in evaluation 
through the assessment reports that it sends to the deputy heads of 
departments and agencies, who are responsible for the evaluation 
function in their respective organizations. 


1.49 We define capacity as sufficient qualified evaluation staff and 
funding to meet needs for effectiveness evaluation. We examined 
staffing and funding for program evaluation in the six departments 
over the five-year period audited to determine whether they were able 
to hire enough staff and to address areas for improvement. 


1.50 We found that, despite having increased funding and staffing, 
the audited departments found it challenging to hire enough qualified, 
experienced evaluation staff to meet needs for effectiveness 
evaluation, and they had not been able to regularly address areas 

for improvement. 


Funding increases have enhanced departments’ capacity for evaluation 


1.51 Evaluation unit funding increased over the audit period 
in all departments audited except Fisheries and Oceans Canada 
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(Exhibit 1.6). This includes funding from the Treasury Board that 

is intended to enable departments to implement the 2001 Evaluation 
Policy, and funding provided following the enactment of the Federal 
Accountability Act for the evaluation of all ongoing grant and 
contribution programs. 


Exhibit 1.6 Evaluation unit funding increased in most departments 


Unit funding in 
2008-09 


Unit funding in 
2004-05 


Department 


Agriculture and Agri-Food Canada $1,029,000 | $1,894,000 
Canadian Heritage | $2,499,000 $3,123,000 
Citizenship and Immigration Canada $650,000 | $1,987,000 
Environment Canada is $732,000 $1,383,000 
Fisheries and Oceans Canada $1,248,000 | $1,162,000 
Human Resources and Skills Development $10,750,000 $13,924,000 
Canada 


Source: Data collection forms completed by departments 


Shortage of experienced evaluators continues 


152 The shortage of experienced program evaluators in the federal 
government is a long-standing concern. It has been noted in past 
Office of the Auditor General audits and in diagnostic studies by the 
Treasury Board of Canada Secretariat, and it was the subject of recent 
discussions within the federal evaluation community. A 2005 report by 
the Secretariat Centre of Excellence for Evaluation stated that “[t]he 
scarcity of evaluation personnel is probably the number one issue 
facing Heads of Evaluation.” 


1.53 We found that during the period covered by this audit, the number 
of professional staff working in the evaluation units in the 

six departments had increased substantially (Exhibit 1.7). This trend was 
also evident in our analysis of descriptive data provided by the remaining 
large federal departments and agencies, which we did not audit. 


1.54 According to officials in the six departments, despite these 
increases in both funding and staff, it remains a challenge to find 
experienced evaluators, particularly at the senior levels. In their view, 
the shortage of experienced evaluators has affected their ability to hire 
the people they need. For example, in one collective staffing process, 
the pool of experienced evaluators was depleted before the demand 
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was met. They also indicated that the shortage of experienced 
evaluators has led to evaluators being hired away by other federal 
evaluation units. 
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Exhibit 1.7 The estimated number of evaluation unit professional staff increased in each department 
between the 2004-05 and 2008-09 fiscal years 


| Number of professional staff | Number of professional staff 
Department in 2004—05* | in 2008-09* 
Agriculture and Agri-Food 6.0 11.0 
Canada 
Canadian Heritage 8.1 Va30. 
(T=. 
Citizenship and Immigration 3.3 i235 
Canada 
Environment Canada 4.0 10.0 
Fisheries and Oceans Canada 4.0 7.0 
Human Resources and Skills: 44.5 54.0 
Development Canada 
Other large departments 176;3 2967 


* Full-time equivalents 


Source: Data collection forms completed by departments 


Evaluator competencies are not adequately defined 


1.55 In diagnostic studies carried out by the Secretariat in 2005, 
deputy heads of departments identified the shortage of qualified 
evaluators as contributing to the inconsistent quality of evaluations. 
Competency profiles can help address this gap, by identifying training 
and development needs and by informing staffing efforts. We examined 
whether the six departments had addressed the challenge of finding 
the right people to do the work, by developing competency profiles for 
their evaluation unit staff. 


1.56 Officials told us that they had begun to develop competency 
profiles. However, these efforts were discontinued, while the Secretariat 
undertook related work that recently resulted in a draft competency 
profile. The development of these profiles has been hindered by the lack 
of agreement about the required competencies of evaluators. 
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Other responsibilities of evaluation units put pressure on capacity 


1.57 According to the Secretariat, evaluation units have assumed a 
number of responsibilities in addition to the traditional evaluation 
studies, including 


° preparing evaluation planning reports and assessments, 


¢ developing results-based management accountability frameworks, 
and 


¢ providing advice and training to program managers on evaluation 
and performance measurement. 


1.58 . We interviewed officials in the six audited departments to find 
out how much time is spent conducting effectiveness evaluations. We 
found that Environment Canada is the only department that conducts 
formal time recording of evaluation unit tasks. It was also the only 
department that was able to provide us with detailed data, from which 
we determined that about 40 percent of its time was spent on tasks 
other than evaluation. 


1.59 The estimates provided by the other departments indicated that 
they spent about the same amount of time on such tasks. While we 
recognize the potential value and importance of these other tasks, they 
nevertheless have an impact on the capacity of evaluation units to 
meet identified needs for effectiveness evaluation. 


Departments use contractors extensively 


1.60 Another diagnostic study reported that a sample of deputy 
heads thought that evaluation units could make a more significant 
contribution and could strengthen staff capacity, by conducting more 
evaluations in-house. According to this study, deputy heads were 
concerned that contracting out evaluation studies prevents evaluation 
units from becoming their department’s subject matter experts, since 
much of what is learned remains with the contractor and not with 

the unit. 


1.61 We looked at the evaluations conducted by the six audited 
departments during the audited period, between the 2004-05 

and 2008-09 fiscal years, to determine whether they were conducted 
by contractors, in-house employees, or both. While about 90 percent 
of the evaluations were wholly or partially conducted by contractors 
(Exhibit 1.8), this varied among the audited departments. In Fisheries 
and Oceans Canada, for example, the figure was 37 percent, while in 
other departments the figures were close to 100 percent. This pattern 
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was also evident in our analysis of descriptive data provided by the 
remaining large federal departments and agencies, which we did 
not audit. 
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Exhibit 1.8 Evaluations conducted by contractors in whole or in part in the audited departments 


Number of evaluations 


200 


150 


al Contractors* 


| | In-house 


100 


90 


2004-05 2005-06 


2006-07 2007-08 
Fiscal years 


2008-09 Total 


*Contractors’ data includes evaluations they conducted both in whole or in part. 


162 Although officials recognized the value of developing in-house 
capacity, they also informed us that they required contractors for 
specific technical or subject matter expertise that was not feasible to 
maintain in-house. 


1.63 The Treasury Board of Canada Secretariat has described 
oversight as one of its central agency roles. Oversight includes policy 
development, monitoring, and reporting on management and 
budgetary performance within government. The Secretariat is 
responsible for the oversight of management policy development and 
the financial oversight of expenditure management. The Secretariat’s 
other central agency roles are leadership in setting the agenda for 
management, and helping departments and agencies improve their 
performance. 


1.64 Under the 2001 Evaluation Policy, the Secretariat was required 
to provide central direction for the evaluation function by 


* establishing a Centre of Excellence for Evaluation to provide 
leadership, guidance, and support to the practice of evaluation; 
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* using evaluation results, where appropriate, in decision making at 
the Centre; 


* setting standards; and © 


* monitoring evaluation capacity in the government. 


The Secretariat has identified a number of improvements in evaluation 


1.65 We expected the Secretariat to support government-wide 
evaluation practices by identifying needed improvements and 
determining and carrying out actions required of the Secretariat, to 
help to ensure that departments and agencies have the tools they need 
to achieve the desired results. 


1.66 In 2004, the Secretariat did an interim evaluation of the 2001 
Evaluation Policy and found gaps in budget and human resources. This 
evaluation called for the Centre of Excellence for Evaluation to play a 
leadership role in helping the evaluation community, by 


¢ advocating the importance of evaluation to senior managers for 
decision making; 


* continuing to help with system-wide capacity building; 


* continuing to develop training, tools, and guides to support policy 
implementation; and 


¢ identifying best practices for the evaluation community. 


1.67 Between.the 2004—05 and 2006-07 fiscal years, the Secretariat 
also carried out several diagnostic studies that included interviews with 
deputy heads, clients, and stakeholders, as well as inquiries into the 
professionalism and the overall role of evaluation in the federal 
government. These studies identified necessary improvements. 


1.68 We noted a number of specific initiatives related to evaluation 
and diagnostic studies, including new guidance and help given to 
departments for recruiting and training evaluators. 


The Secretariat carried out extensive monitoring of the evaluation function 


1.69 By the 2004-05 fiscal year, the Secretariat had developed several 
monitoring tools to collect information about the evaluation function 
across government. Secretariat officials identified similar monitoring 
activities in 2009: 


* annual capacity assessment survey; 
¢ Evaluation Information Resource Component—a database; 


* periodic in-depth review of the quality of evaluation reports; 
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* ongoing reviews of evaluations, results-based management 
accountability frameworks, and departmental evaluation plans; 


¢ department and agency visits and interaction; 
° feedback to individual entities: 


* communication of best practices in evaluation methodology and 
reporting, and in managing the evaluation function; 


* review of the evaluation content of departmental Estimates 
documents (Report on Plans and Priorities and the Departmental 
Performance Report); and 


¢ Management Accountability Framework assessments. 


1.70 The Secretariat describes the Management Accountability 
Framework as one of several tools it uses to assess management 
performance in departments. Although we did not audit the 
Framework, we noted that the Secretariat has used it in several ways, 
including changing the Framework itself, for example, by refining the 
assessment ratings. Program sector analysts and analysts at the Centre 


-of Excellence for Evaluation also cited the knowledge gained from the 


Framework’s assessments as potentially helpful, as it could be used to 
improve guidance and tools, for example, in the development of the 
Standard on Evaluation that was issued with the 2009 Policy on 
Evaluation. However, we also noted that, because this knowledge was 
not always documented, its impact was not always clear. 


171 The Secretariat’s first (and, to date, only) published report on 
the evaluation function—The Health of the Evaluation Function in 
the Government of Canada Report for Fiscal Year 2004-05 
based on the first of the annual capacity assessment surveys. These 
surveys are used to collect information on evaluation infrastructure, 
resources, production, and results and, for larger departments, to 
collect information on evaluation planning and resource requirements. 
The Secretariat has conducted the capacity assessment survey every 
year since the 2004-05 fiscal year. 


Was 


1.72 The diagnostic studies, the Health of the Evaluation Function 
Report, and ongoing monitoring provided an analytical base for the 
first Government of Canada Evaluation Plan, 2005-06, which was 
developed in September 2005. Once again, this was the only one of 
its kind; no other such plan appeared in the period ending 


31 March 2009. 


1.73 The Secretariat carried out extensive monitoring. The value of 


. . _ 2 ail a 
this monitoring, of annually reporting on the evaluation function's 
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health (see paragraph 1.89), and of developing a government-wide 
plan, were recognized as requirements in the 2009 Policy on 
Evaluation, as well as in earlier drafts of the policy. However, over the 
period of our audit, the Secretariat developed only one such report and 
plan. For the purposes of continuous improvement, it will be important 
for the Secretariat to pursue these oversight activities, with 
departments, on a regular and systematic basis. 


Sustained support for effectiveness evaluation is lacking 


1.74 The Treasury Board’s 2001 Policy on Evaluation advised that 
evaluations consider program relevance, success, and cost- 
effectiveness. We therefore expected the Treasury Board of Canada 
Secretariat to provide guidance to departments and to help them 
identify these elements and improve effectiveness evaluation. 


1.75 One area the Secretariat has clearly identified is the need to 
improve the use of evaluation for expenditure review. In 2005, a 
discussion document from the President of the. Treasury Board, 
Management in the Government of Canada: A Commitment to Continuous 
Improvement, expressed concern that evaluation must become more 
directly linked to decisions regarding resource allocation. 


176 In our audit of the expenditure management system (EMS) 
(November 2006 Report of the Auditor General of Canada, 

Chapter 1—Expenditure Management System at the Government 
Centre), we recommended a systematic review of the relevance and 
value-for-money of ongoing programs. The Public Accounts Committee 
pursued this issue in 2008 by recommending that the Secretariat develop 
an action plan to hire and train the necessary evaluators and that it 
reinforce the importance of evaluation as a key requirement in the EMS. 


1.77. The government accepted these recommendations and launched 
the new expenditure management system that included strategic review. 
It viewed evaluation as the primary source of neutral and systematic 
information on the ongoing relevance and performance of polices and 
programs. In light of these developments, we looked for the Secretariat’s 
actions and initiatives in support of effectiveness evaluation. 


178 We noted an initiative to develop a value-for-money tool for 
evaluation, launched as a pilot project in 2006. However, this initiative 
came to an abrupt halt in 2008 and, over the period of our audit, did 
not move beyond the pilot stage. The Secretariat found that the tool 
was being used to meet minimum data requirements, rather than the 
new policy’s objective of examining effectiveness issues. As we 
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completed our audit, Secretariat officials informed us that the project 
is continuing its development and is being revised to take into 
consideration the requirements of the new policy on evaluation. 


179 The Centre of Excellence for Evaluation has undertaken a 
number of similar initiatives related to capacity development, but 
during our audit, we found that the results of these initiatives are 
unclear. We noted policy requirements and related initiatives for 
evaluators to support the measurement of program performance. 
However, we did not find that the Secretariat had made progress in 
developing tools to help departments address the long-standing 
problem of insufficient data for the evaluation of effectiveness. 


180 While the Secretariat recognized the value of effectiveness 
evaluation, particularly for expenditure management, it did not issue, 
over the period of our audit, adequate guidance or tools to support 
effectiveness evaluation. 


181 The renewal of the 2001 Policy on Evaluation took far longer 
than expected. We found documentation from 2006, 2007, and 2008 
indicating the policy would be completed in each of these years. In 
fact, the new policy took effect only on 1 April 2009. However, at the 
time of our audit, despite the three years that were spent developing 
the policy, the Secretariat had not issued guidance to departments on 
its implementation. 


1.82 Recommendation. In developing tools, guidance, and support 
for departments, the Treasury Board of Canada Secretariat should 
regularly identify gaps that it needs to act on, develop plans to address 
these gaps, and act on these plans. 


The Treasury Board of Canada Secretariat’s response. Agreed. 

The Secretariat is presently developing guidance for departments and 
agencies to support the implementation of the new Treasury Board 
Policy on Evaluation (April 2009). Guidance in a number of key areas, 
including departmental evaluation planning and performance 
measurement strategies, is expected to be available to departments and 
agencies by 31 March 2010. Further guidance will be issued over the 
course of the 2010-11 fiscal year. 


During the past two years, the Treasury Board of Canada Secretariat 
has issued guidance on the development of performance measurement 
frameworks required under the 2005 Policy on Management 
Resources and Results Structures, which was not covered by the 
current audit. In addition, through its annual Management 
Accountability Framework assessments, the Secretariat provides 
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advice and support to departments on the quality and use of 
evaluation as well as on the quality of performance measurement 
frameworks associated with departmental Management, Resources 
and Results Structures. 


The Secretariat continually consults with departments and agencies 
on their needs and monitors policy implementation to identify 
weaknesses within the government-wide evaluation function. Where 
the Secretariat determines a need to develop further tools, guidance, 
or other supports, it includes these activities in its business plans. 


Oversight and support require experienced staff 


1.83 We examined whether the Treasury Board of Canada Secretariat 
had the human and financial resources needed for government-wide 
oversight and support of the program evaluation function. 


1.84 The Centre of Excellence for Evaluation’s major responsibilities 
include 


¢ renewing the evaluation policy; 


* acting as a policy centre, by providing advice and analysis to 
: support the expenditure management system; 


¢ conducting Management Accountability Framework assessments 
of the evaluation function; and 


¢ providing support for the evaluation function across government. 


1.85 In addition, the Centre monitors the evaluation policy. 

Many of these tasks are analytical and call for experienced personnel. 
The review of evaluations, program accountability frameworks, and 
submissions requires sufficient expertise to provide recommendations 
and guidance to program sector analysts as well as to the Treasury 


Board itself. 


1.86 We compiled information on the Centre’s workload and 
compared it to the resources allocated over the audit period. The 
funding for oversight work is largely salary-based. For the period of our 
audit, the staff complement of the Centre varied in size, from only 8 in 
the 2005—06 and 2006-07 fiscal years to 12 in the 2008-09 fiscal year. 
Overall, the staff levels during the 2005-06 to 2008-09 period were 
lower than the 15 on staff in the 2004-05 fiscal year, even though 

the Centre workload pertaining to its oversight activities was . 
increasing. For example, the number of Treasury Board submissions 
reviewed by the Centre almost doubled from the 2004-05 to 2008-09 


fiscal years. 
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1.87 The Secretariat clearly requires experienced analysts with 
appropriate expertise in evaluation in order to meet workload 
demands. The limited number of staff allocated to these functions 
during the audited period may have contributed to the lack of 
sustained support for effectiveness evaluation. 


188 Recommendation. The Treasury Board of Canada Secretariat 
should ensure that it allocates sufficient resources to tasks that require 
evaluation expertise. 


The Treasury Board of Canada Secretariat’s response. Agreed. 

In renewing the Policy on Evaluation, the Government has 
strengthened its commitment to evaluating the value for money of 
federal programs and reaffirmed the Secretariat’s role of leading the 
evaluation function. In performing its functional leadership role 
established in the new policy, the Treasury Board of Canada Secretariat 
will ensure that the resources necessary for performing this role are 
considered and sufficient resources are allocated at the Secretariat to 
tasks that require evaluation expertise. 


Care is needed in the implementation of the new coverage requirements 


1.89 By 2006, the Secretariat had completed its diagnostic work and 
issued its Health of the Evaluation Function Report. The Secretariat knew 
the challenges for the evaluation function associated with the legal 
requirement that all ongoing grant and contribution programs (a part 
of direct program spending) be evaluated, and that departments were 
hard pressed to meet it. The requirement was especially challenging for 
the audited departments that have a high proportion of such programs, 
but additional central funding was provided in 2007 to support 
meeting this coverage requirement. In particular, as the Secretariat 
itself noted, requiring full coverage made it more difficult to target 
evaluation efforts on the basis of risks. When the legal requirement 
was enacted in 2006, it created pressure for 100 percent coverage of all 
direct program spending, in parallel with the later reform of the 
expenditure management system. 


1.90 We noted that deputy heads of departments remain responsible 
for implementing the 2009 evaluation policy, including the expanded 
coverage requirements. It will be important for the Secretariat to work 
with departments to ensure that they are fully prepared to implement 
these coverage requirements, in order to meet the expectations set out 


in the new evaluation policy. 
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1.91 The implementation of the new coverage requirement faces 
serious challenges. Earlier requirements for full coverage were never 
met. Current legal requirements for effectiveness evaluation of all 
grants and contributions programs have been difficult to meet, and 
department officials told us that they have concerns about their 
capacity to respond to these requirements. Moreover, we found a 
shortage of experienced evaluators and extensive use of contractors | 
over the period audited. For example, Environment Canada estimated 
that it would have to double the complement of its evaluation unit 
over the next four years, or sacrifice evaluation depth in order to 
achieve full coverage. 


1.92 In our view, it will be important for the Secretariat and 
departments to carry out effectiveness evaluation of programs that are 
susceptible to significant change because of shifting priorities and 
circumstances. These are programs where evaluations of the relevance, 
impact, and achievement of objectives can be put to best use. During 
the transition to full coverage, these programs may present the biggest 
opportunities for effectiveness evaluation. 


1.93 Recommendation. The Treasury Board of Canada Secretariat 
should help departments prepare to implement the new coverage 
requirements. During the transition period, the Secretariat should 
provide advice and guidance for effectiveness evaluation, focusing on 
programs where such evaluation can be put to best use. 


The Treasury Board of Canada Secretariat’s response. Agreed. The 
Secretariat is planning to issue written guidance for making risk-based 
choices for evaluation coverage to support departments during the 
transition period. This guidance is expected by 31 March 2010. 


Throughout the transition period, the Secretariat will also help 
departments prepare to implement the new coverage requirements 
that come into effect after 31 March 2013. The Secretariat will 
provide leadership in the development and sharing of effective 
evaluation practices across departments, as well as support capacity- 
building initiatives in the evaluation function government-wide. 


Conclusion 


1.94 The six departments included in this audit—Agriculture and 
Agri-food Canada, Canadian Heritage, Citizenship and Immigration 
Canada, Environment Canada, Fisheries and Oceans Canada, and 
Human Resources and Skills Development Canada—followed 
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systematic processes to plan their effectiveness evaluations. As well, 
most planned evaluations were completed. However, the evaluations 
that were conducted by these departments only covered a low 
proportion of overall departmental expenses, and most of the 
evaluations we examined were hampered by inadequate data. As a 
result, departments were not able to demonstrate that they were 
sufficiently meeting needs for effectiveness evaluation. 


1.95 Based ona sample of effectiveness evaluations, we found that the 
audited departments often did not have the necessary performance 
information to evaluate whether programs are effective. 


196 Moreover, with the exception of Environment Canada, which 
has processes in place to identify needed improvements, the audited 
departments did not demonstrate that they had regularly identified 
and addressed areas for improvement in effectiveness evaluation 
during the audit period. Such a cycle of continuous improvement 
would steadily add value to effectiveness evaluation. 


1.97 The departments we examined expressed concerns about their 
capacity to implement evaluation of all direct program spending, as 
required under the 2009 Policy on Evaluation. Even before these 
expanded requirements, they found it challenging to hire enough 
experienced evaluators to fully meet needs for effectiveness evaluation, 
and they had not been able to regularly address areas for improvement. 
In our view, identifying programs where effectiveness information can 
be put to the best use will be a key part of implementing the coverage 
requirements of this policy. 


1.98 Over the past five years, the Treasury Board of Canada 
Secretariat has introduced initiatives to address improvements in 
evaluation. However, support for effectiveness evaluation, which is an 
important area for the Secretariat, did not receive sustained attention. 
While the Secretariat did regularly identify areas for improvement, 

it did not provide adequate guidance. 


1.99 Overall, we found that, in the six departments we audited, 
needs for effectiveness evaluation were not being adequately met. 
Improvements are required in departments, and in the oversight and 
support activities of Treasury Board of Canada Secretariat, in order to 


remedy the situation. 


1.100 These findings are similar to many reported by the Office in 
previous audits of program evaluation. Taken together, they raise basic 
questions about effectiveness evaluation in the federal government. 
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1.101 In our view, the federal evaluation function is at a crossroads. 

A vital public purpose is served when effectiveness evaluation informs 
the important decisions that Canadians are facing. Departments face 
greater expectations than ever before and are taking on added 
responsibilities. Much remains to be done to meet the challenge. 
Continuous improvement is the way forward. 
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About the Audit 


All of the audit work in this chapter was conducted in accordance with the standards for assurance 
engagements set by The Canadian Institute of Chartered Accountants. While the Office adopts these 
standards as the minimum requirement for our audits, we also draw upon the standards and practices 
of other disciplines. 


Objectives 


The overall objective of this audit was to determine whether selected departments and the Treasury Board 
of Canada Secretariat are meeting the needs for effectiveness evaluation and are identifying and making 
improvements in effectiveness evaluation. 


The audit objectives for the three lines of enquiry were as follows: 


- Determine whether selected departments can demonstrate that they are meeting needs 
for effectiveness evaluation and regularly identify and address areas for improvement. 


¢ Determine whether selected departments can demonstrate that they have the capacity to meet 
key needs for effectiveness evaluation and regularly identify and address areas for improvement. 


¢ Determine whether the Treasury Board of Canada Secretariat’s government-wide oversight of 
the program evaluation function has regularly identified and addressed areas for improvement 
that ensure that departments have the capacity to meet needs for effectiveness evaluation. 


Scope and approach 


Focus on effectiveness evaluation. Based on the Auditor General Act, section 7(2)(e), this audit 
examined program evaluation government-wide, in relation to the measurement of program effectiveness 
(meaning the assessment of the extent to which programs are relevant, produce impacts and outcomes, 
achieve their objectives, and are cost effective). Other types of evaluation examine program 
implementation and management. 


Not a compliance audit. Because the 2001 Treasury Board evaluation policy was replaced in April 2009, 
the audit did not focus on compliance with policy. 


Evaluation quality. We examined evaluation quality by determining whether departments had processes 
in place, including quality assurance, to ensure that their effectiveness evaluations are appropriate for their 


intended uses. 


Strategic review and the Expenditure Management System. In view of the link between the Expenditure 
Management System and evaluation (that is, evaluation is seen as a key source of information for strategic 
review), the audit sought to determine whether evaluations met the need created by strategic review. 
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Selection of entities. The entities selected for examination in the audit were the Treasury Board of 
Canada Secretariat and the following six departments: 


« Agriculture and Agri-Food Canada, 

« Canadian Heritage, 

¢ Citizenship and Immigration Canada, 
« Environment Canada, 

- Fisheries and Oceans Canada, and 


« Human Resources and Skills Development Canada. 


The selection of departments was based on factors such as materiality, range of program types, nature of 
the evaluation function, and management accountability framework (MAF) ratings and whether a 
strategic review had been carried out. Our audit also included descriptive information provided by other 
large departments (those participating in the annual MAF process) who attested to the accuracy of 
information they provided. 


During our audit, we conducted interviews, reviewed files and documents, analyzed descriptive 
information provided and attested to by large departments, and met with focus groups who provided us 
with an informed stakeholder perspective. 


Period covered by the audit 


The audit period was between the 2004-05 and 2008-09 fiscal years. This period was chosen because 
enough time would have elapsed since the introduction of the 2001 policy to allow its effects to take hold. 
In addition, the 2004-05 fiscal year was when the Treasury Board of Canada Secretariat carried out an 
interim evaluation of the 2001 policy. It was also the year when the Secretariat engaged contractors to 
complete a series of diagnostic studies aimed at understanding the “state of play” of the function at that 
time. Focusing on the period between the 2005—06 and 2008-09 fiscal years allowed us to examine the 
impact of the Secretariat’s efforts to understand how well the 2001 policy was working, and to examine 
how the Secretariat identified problems between the 2002-03 and 2004-05 fiscal years and addressed 
these problems between the 2004—05 and 2008-09 fiscal years. 


Audit work for this chapter was substantially completed on 31 May 2009. 
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Criteria 


Listed below are the criteria that were used to conduct this audit and their sources. 


Criteria - 


) 


Sources 


We expected that departments could demonstrate that program 
evaluation plans take appropriate account of needs for 
effectiveness evaluation. 


Auditor General Act, section 7(2)(e) 
Financial Administration Act, section 42(1) 
Treasury Board Evaluation Policy (2001) 


Treasury Board of Canada Secretariat, Centre of Excellence for 
Evaluation, Evaluation Function in the Government of Canada 
(2004), Appendix 2, Evaluation Standards in the Government 
of Canada 


Treasury Board of Canada Secretariat, Centre of Excellence 
for Evaluation, A Guide to Developing a Risk-Based 
Departmental Evaluation Plan (2005), section 3.2 


Treasury Board Policy on Transfer Payments (2008) 


We expected that departments could demonstrate that they 
have acted on program evaluation plans to meet key needs. 


Treasury Board Evaluation Policy (2001) 


Treasury Board of Canada Secretariat, Centre of Excellence 
for Evaluation, A Guide to Developing a Risk-Based 
Departmental Evaluation Plan (2005), page 14 


We expected that departments could demonstrate that their 
effectiveness evaluations appropriately meet identified needs. 


Treasury Board Evaluation Policy (2001) 


Treasury Board of Canada Secretariat, Centre of Excellence for 
Evaluation, Evaluation Function in the Government of Canada 
(2004), Appendix 2, Evaluation Standards in the Government 
of Canada 


We expected that departments could demonstrate that they 
regularly identify and act on required improvements in meeting 
needs for effectiveness evaluation. 


Treasury Board Evaluation Policy (2001) 


Treasury Board of Canada Secretariat, Centre of Excellence for 
Evaluation, Evaluation Function in the Government of Canada 
(2004), Appendix 2, Evaluation Standards in the Government 
of Canada 


We expected that departments could demonstrate reasonable 
efforts to ensure sufficient qualified evaluation staff to meet key 
needs for effectiveness evaluation. 


Treasury Board Evaluation Policy (2001) 


Treasury Board of Canada Secretariat, Centre of Excellence for 
Evaluation, Evaluation Function in the Government of Canada 
(2004), Appendix 2, Evaluation Standards in the Government 
of Canada 


Treasury Board of Canada Secretariat, People Component of 
the Management Accountability Framework (PCMAF) (2005), 
page 1 


Public Service Commission of Canada, Staffing Management 
Accountability Framework (SMAF) (2005), page 4 


Government Response to the Fourth Report on the Standing 
Committee on Public Accounts: The Expenditure Management 
System at the Government Centre and the Expenditure 
Management System in Departments (2008), page 8 
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Criteria 


he Sane = 


Sources 
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We expected departments could demonstrate that the amount 
and the time frame of funding for effectiveness evaluation meet 
key needs. ° 


e Treasury Board of Canada Secretariat, Centre of Excellence 
for Evaluation, A Guide to Developing a Risk-Based 
Departmental Evaluation Plan (2005), pages 5 and 13 


We expected that departments could demonstrate that 
evaluators have sufficient independence from program managers 
and that their objectivity is not hindered. 


* Treasury Board Evaluation Policy (2001) 


¢ Treasury Board of Canada Secretariat, Centre of Excellence for 
Evaluation, Evaluation Function in the Government of Canada 
(2004), Appendix 2, Evaluation Standards in the Government 
of Canada 


We expected that departments could demonstrate that they 
regularly identify and act on required improvements to capacity 
to meet needs for effectiveness evaluation. 


* Treasury Board Evaluation Policy (2001) 


e Public Service Commission of Canada, Staffing Management 
Accountability Framework (SMAF), page 5 


We expected that the Treasury Board of Canada Secretariat has 
the resources required for government-wide oversight of the 
program evaluation function. 


Financial Administration Act, section 6(7) 


We expected that the Treasury Board of Canada Secretariat 
could support the practice of government-wide evaluation by 
identifying needed improvements and determining and carrying 
out actions required of the Secretariat to help ensure that 
departments and agencies have the tools they need to achieve 
the desired results. 


Treasury Board Evaluation Policy (2001) 


The Standing Committee on Public Accounts, Report on the 
Expenditure Management System at the Government Centre 
and the Expenditure Management System in Departments 
(2008), page 16 


Management reviewed and accepted the suitability of the criteria used in the audit. 


Audit team 


Assistant Auditor General: Neil Maxwell 
Principal: Tom Wileman 
Lead Director: Colin Meredith 
Directors: Doreen Deveen 

Leslie Levita 


Irene Andayo 

Helene Charest 

Jeff Graham 

Krista Hilge 
Chandrawattie Samaroo 


Jj 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 1. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


Meeting needs for effectiveness evaluation 


1.37 Agriculture and Agri-food Agriculture and Agri-Food Canada’s response. Agreed. The 
Canada, Canadian Heritage, Department agrees that the systematic collection of program 
Citizenship and Immigration Canada, performance data by managers is necessary to report on program 
Environment Canada, Fisheries and performance and to support effectiveness evaluations. 


Oceans Canada, and Human Resources 
and Skills Development Canada should 
develop and implement action plans to 
ensure that ongoing program 

performance information is collected to 


support effectiveness evaluation. 
(1.14-1.36) The Department’s evaluation function reviews program 


performance measurement strategies as they are developed to 


As required by the Treasury Board Policy on Transfer Payments, 
a performance measurement strategy for ongoing management of 
transfer payment programs, including performance measures and 
indicators and a data collection strategy, is developed for each 
new transfer payment program. 


ensure that outcomes are defined, measurable, and attributable 
and that the strategies, if implemented, are sufficient to support 
future evaluation work. 


Beginning in the 2009-10 fiscal year, the Department will 
conduct annual state of performance measurement reviews. The 
first such review will assess performance measurement practices 
at the Department and the adequacy of data collected for 
programs soon to be evaluated. An action plan to address its 
recommendations will be developed and its implementation 
monitored with a view to strengthening the Department's 
practices in this area. 


Canadian Heritage’s response. Agreed. The Department is 
executing its Action Plan for Implementation of the 
Management, Resources and Results Structures (MRRS) Policy 
to ensure that program staff are able to fulfill their responsibility 
for developing and maintaining performance measurement 
strategies. Action plan measures include 


¢ the provision of information sessions and workshops, 
* the establishment of indicators and corresponding targets, 


* the development of robust methodologies to demonstrate 


outcomes, 
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° the establishment of a centre of expertise on performance 
measurement within the Department, 


° the design and implementation of adequate tools and 
guidelines, 


¢ the establishment of relevant information technology at 
systems, and 


¢ the regular analysis and reporting of collected data. 


This action plan is expected to be completed by end a 
the 2011-12 fiscal year. 


The Department’s Office of the Chief Audit and Evaluation 
Executive is continually providing advice and support to 
Department managers in their efforts to implement this action 
plan. In line with the Transfer Payment Policy, this office is also 
providing timely advice and support on program design and 
performance measurement strategies through the review of 
official approval documents for the creation and renewal of new 
or existing programs. Finally, as required under the new 
Evaluation Policy, the office will be submitting, in the 2010-11 
fiscal year, its first annual report to the departmental evaluation 
committee on the state of performance measurement in the 
Department. 


Citizenship and Immigration Canada’s response. Agreed. The 
Department recognizes the value of ongoing performance 
information for evaluation and will continue to support related 
departmental activities. The Department will develop an action 
plan that includes the renewal of the comprehensive 
departmental performance measurement framework and 
program activity architecture, and furthers the integration of the 
Framework into the business planning process. 


Environment Canada’s response. Agreed. The Department 
accepts this recommendation, which echoes the intent of 

the 2009 Evaluation Policy related to performance information. 
As such, actions are already under way within Environment 
Canada to implement this recommendation. These include 
ongoing monitoring of the implementation of management 
responses to previous evaluations that have identified concerns 
with performance information, and the development and 
implementation of a strategy to inform all department managers 
of the Evaluation Policy requirements pertaining to performance 
measurement. 
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In addition, the Department's evaluation plan will be expanded 
to include a monitoring component to verify, within available 
resources, the status of performance data collected in the 
department and whether sufficient performance information will 
be available to support upcoming evaluations. This monitoring 
component will be included in the 2010-15 Evaluation Plan and 
will be updated annually thereafter. 


Further, for the 2010-11 fiscal year, the Department's 
performance measurement framework has been linked to the 
Department’s program activity architecture, in that performance 
measures have been identified for all programs. 


Fisheries and Oceans Canada’s response. Agreed. The 
Department’s performance measurement framework links its 
core indicators to the departmental program activity 
architecture (PAA), thus identifying performance measures for 
all program activities and sub-activities. Each fiscal year, the 
Department conducts an analysis of the state of performance 
measurement in the Department and provides an annual report 
to the Departmental Evaluation Committee. In addition, the 
Department will develop and implement an action plan to 
ensure that ongoing program performance information is 
collected to support effectiveness evaluation by the end of 


August 2010. 


Human Resources and Skills Development Canada’s 
response. Agreed. The Department accepts this 
recommendation that echoes the intent of various Treasury 
Board of Canada Secretariat policies, including the 2009 
Evaluation Policy. As such, the Department already gathers and 
monitors ongoing performance information to support 
effectiveness evaluation, including 


* monitoring implementation of management responses to 
previous evaluations that have identified data issues for 
current programs; 


* undertaking early evaluative work in advance of the formal 
initiation of effectiveness evaluations as part of the evaluation 
planning process, to review the state of performance data and 
logic models; and 


* monitoring the implementation of new programs (for example, 
Economic Action Plan initiatives) to ensure the necessary 
administrative and performance data are available to support 


future evaluation activities. 
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143 Agriculture and Agri-food 
Canada,-Canadian Heritage, 
Citizenship and Immigration Canada, 
Environment Canada, Fisheries and 
Oceans Canada, and Human Resources 
and Skills Development Canada should 
consider the merits of including 
external experts on their departmental 
evaluation committees. The Treasury 
Board of Canada Secretariat should 


provide guidance to departments in this 
regard. (1.38-1.42) 


The Department is also undertaking a comprehensive review, 
refinement, and validation of its Performance Measurement 
Framework (PMF) to make it more robust and comprehensive, 
to support ongoing planning, monitoring, and managing for 
results and the Evaluation Directorate has been actively 
involved in this work. The progress made on the departmental 
PMF will support both performance monitoring as well as 
evaluation of relevance and effectiveness. 


Agriculture and Agri-Food Canada’s response. Agreed. The 
Department has recently introduced a number of practices to 
ensure production of strong evaluation reports. They include 


seeking input from external experts on evaluations in progress. 


The Department will also consider including external members 
on its Departmental Evaluation Committee, in the’context of 
guidance provided by Treasury Board of Canada Secretariat. 


Canadian Heritage’s response. Agreed. At the Department, 
oversight of the evaluation function is provided by the Strategic 
Policy, Planning and Evaluation Committee (SPPEC), chaired by 
the Deputy Head. This structure provides opportunities for 
enhanced integration between the policy, planning, and 
evaluation functions of the department. As well, the 
Department already brings key evaluation reports when 
necessary to its department audit committee for review and 
discussion. The Department will collaborate with the Treasury 
Board of Canada Secretariat's Centre of Excellence in 
Evaluation to assess the value added of integrating the advice of 
external evaluation experts to inform the work of departmental 
evaluation committees. 


Citizenship and Immigration Canada’s response. Agreed. The 
Department will assess the benefits of including an external 
evaluation expert on the departmental evaluation committee. 


Environment Canada’s response. Agreed. The Department 
accepts the recommendation and will await guidance from the 
Treasury Board of Canada Secretariat on the inclusion of 
external members on departmental evaluation committees. 


Fisheries and Oceans Canada’s response. Agreed. The 
Department will consider the merit of including external 
members on its department evaluation committee, in the 
context of guidance provided by the Treasury Board of Canada 
Secretariat. 
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1.48 Agriculture and Agri-food 
Canada, Canadian Heritage, 
Citizenship and Immigration Canada, 
Fisheries and Oceans Canada, and 
Human Resources and Skills 
Development Canada should 
implement systematic processes to 
determine whether their effectiveness 
evaluations are meeting government- 
wide requirements and internal 
corporate needs, and act on areas 
identified for improvement. The 
Treasury Board of Canada Secretariat 
should monitor and provide any 

-additional support it considers 
necessary for the implementation of 
these processes. (1.44—1.47) 


Human Resources and Skills Development Canada’s 
response. Agreed. The Department has, in the past, included 
members from outside the Department on its departmental 
evaluation committee. The Department will reconsider the 
formal inclusion of external experts on the current 
Departmental Evaluation Committee and will look to the 
‘Treasury Board of Canada Secretariat for guidance on this part 
of the Evaluation Policy. 


The Treasury Board of Canada Secretariat’s response. 
Agreed. The Secretariat agrees that it should provide guidance 
to departments on the possible merits of including external 


experts on their departmental evaluation committees. These 
actions will be completed by 31 March 2010. 


Agriculture and Agri-Food Canada’s response. Agreed. The 
Department accepts this recommendation and notes that in the 
past year, it has introduced a number of systematic processes, 
which together will ensure effectiveness evaluations address 
senior management information needs in a timely manner. They 
include annual consultations on evaluation priorities, requests 
for feedback on completed evaluations, and annual Head of 
Evaluation reports on the performance of the evaluation 
function. 


Canadian Heritage’s response. Agreed. The Department 
recognizes the need to establish systematic processes to assess 
whether effectiveness evaluations are addressing needs. The 
Evaluation Services Directorate is already developing a 
performance measurement framework and management strategy 
to identify clear performance expectations and standards for the 
Department’s evaluation function. A systematic process to 
collect, analyze, and report on performance data and client 
satisfaction will be implemented in order to identify areas of 
improvements. Based on data collected during the first year of 
implementation (the 2010-11 fiscal year), periodic reporting to 
the Departmental Evaluation Committee should begin by 

the 2011-12 fiscal year. This performance information will 
complement data already reported in the context of the annual 
management accountability framework assessments conducted 
by the Treasury Board of Canada Secretariat. 
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Citizenship and Immigration Canada’s response. Agreed. The 
Department will assess the need for systematic processes in 
addition to the Management Accountability Framework 
assessment process, the oversight of the Departmental 
Evaluation Committee, requirements for annual reporting, and 
the’ Department’s evaluation process, which includes several 
steps of consultation and feedback from the Department’s 
branches. 


Fisheries and Oceans Canada’s response. Agreed. The 
recommendation echoes the intent of the 2009 Policy on — 
Evaluation. As such, actions are already under way within the 
Department to implement a systematic process to determine 
whether effectiveness evaluations are meeting internal corporate 
needs and government-wide needs (i-e., a strategic review) and 
to act on areas identified for improvement. Further work in this 
area will be completed in the context of guidance provided by 
the Treasury Board of Canada Secretariat. 


Human Resources and Skills Development Canada’s 
response. Agreed. The Department currently employs a variety 
of systematic processes to ensure the quality and relevance of its 
evaluations for both internal and government-wide needs. 
External peer reviewers and evaluation advisory committees are 
a mandatory element of the evaluation work to ensure that 
evaluations are meeting information needs. Further work in 
systematically determining whether evaluations are meeting 
government and senior management needs will be developed, 
building upon the current annual Management Accountability 
Framework assessment process led by the Treasury Board of 
Canada Secretariat. Areas for improvement will be identified 
and reported to the Departmental Evaluation Committee. 


The Treasury Board of Canada Secretariat’s response. Agreed. 
The Secretariat agrees that it should assist departments as 
necessary in their implementation of processes to determine 
whether evaluations are meeting government-wide needs and 
should provide support to departments that it considers necessary. 


The new Treasury Board Policy on Evaluation, which came into 
effect on 1 April 2009, includes specific requirements to 
enhance evaluation coverage and examination of program 
effectiveness as well as performance information to support 
evaluations. The policy calls on the Secretary of the Treasury 
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Board to provide functional leadership for evaluation across 
government, including monitoring and reporting annually to the 
‘Treasury Board on the health of the evaluation function. 


The Secretariat currently carries out a large portion of this work 
through the annual Management Accountability Framework 
assessment process, which the Office of the Auditor General 
acknowledges was not covered by the current audit. The 
Secretariat communicates recommended areas for improvement 
in evaluation through the assessment reports that it sends to the 
deputy heads of departments and agencies, who are responsible 
for the evaluation function in their respective organizations. 


Oversight and support 


- 1.82 In developing tools, guidance, 
and support for departments, the 
Treasury Board of Canada Secretariat 
should regularly identify gaps that it 
needs to act on, develop plans to 


address these gaps, and act on these 
plans. (1.74-1.81) 


The Treasury Board of Canada Secretariat’s response. 
Agreed. The Secretariat is presently developing guidance for 
departments and agencies to support the implementation of the 
new Treasury Board Policy on Evaluation (April 2009). 
Guidance in a number of key areas, including departmental 
evaluation planning and performance measurement strategies, is 
expected to be available to departments and agencies by 

31 March 2010. Further guidance will be issued over the course 
of the 2010-11 fiscal year. 


During the past two years, the Treasury Board of Canada 
Secretariat has issued guidance on the development of 
performance measurement frameworks required under the 


- 2005 Policy on Management Resources and Results Structures, 


which was not covered by the current audit. In addition, 
through its annual Management Accountability Framework 
assessments, the Secretariat provides advice and support to 
departments on the quality and use of evaluation as well as on 
the quality of performance measurement frameworks associated 
with departmental Management, Resources and Results 
Structures. 


The Secretariat continually consults with departments and 
agencies on their needs and monitors policy implementation to 
identify weaknesses within the government-wide evaluation 
function. Where the Secretariat determines a need to develop 
further tools, guidance, or other supports, it includes these 
activities in its business plans. 


Report of the Auditor General of Canada—Fall 2009 


Chapter 1 45 


| EVALUATING THE EFFECTIVENESS OF PROGRAMS 


1.88 The Treasury Board of Canada 


Secretariat should ensure that it 
allocates sufficient resources to tasks 
that require evaluation expertise. 
(1.83-1.87) 


1.93 The Treasury Board of Canada 
Secretariat should help departments 
prepare to implement the new coverage 
requirements. During the transition 
period, the Secretariat should provide 
advice and guidance for effectiveness 
evaluation, focusing on programs where 
such evaluation can be put to best use. 


(1.89-1.92) 


The Treasury Board of Canada Secretariat’s response. 
Agreed. In renewing the Policy on Evaluation, the Government 
has strengthened its commitment to evaluating the value for 
money of federal programs and reaffirmed the Secretariat’s role 
of leading the evaluation function. In performing its functional 
leadership role established in the new policy, the Treasury Board 
of Canada Secretariat will ensure that the resources necessary 
for performing this role are considered and sufficient resources 
are allocated at the Secretariat to tasks that require evaluation 
expertise. 


The Treasury Board of Canada Secretariat’s response. 


Agreed. The Secretariat is planning to issue written guidance for 
making risk-based choices for evaluation coverage to support 


departments during the transition period. This guidance is 
expected by 31 March 2010. 


Throughout the transition period, the Secretariat will also help 
departments prepare to implement the new coverage 
requirements that come into effect after 31 March 2013. The 
Secretariat will provide leadership in the development and 
sharing of effective evaluation practices across departments, as 
well as support capacity-building initiatives in the evaluation 
function government-wide. 
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Main Points 


In Canada, the federal government and the provinces and territories 
share jurisdiction over immigration. Citizenship and Immigration 
Canada (CIC) is generally responsible for the selection of immigrants 
and other foreign nationals and for ensuring that they are admissible— 
that is, that they do not present any risk to the health and safety of 
Canadians. The Department has also signed agreements with most 
provinces and territories allowing them to play an active role in 
selecting immigrants to meet the specific needs of their labour markets. 


In 2008, Canada admitted about 250,000 people as permanent 
residents, including about 150,000 individuals and their immediate 
family members selected on the basis of attributes that would enable 
them to succeed in a dynamic labour market, such as education, 
professional experience, and official language ability. In addition, 
Canada allowed almost 370,000 temporary foreign workers in 2008 to 
fill a short-term need for labour. 


We examined how CIC plans for and manages programs designed to 
facilitate the entry of permanent and temporary workers into Canada 
and the recognition of foreign credentials in Canada. In addition, we 
looked at the role of Human Resources and Skills Development 
Canada (HRSDC) in supporting the planning and delivery of these 
programs, including the issuance of labour market opinions by its 
Service Canada offices. The audit covered the period from June 2002, 
when the Immigration and Refugee Protection Act came into effect, to 


30 June 2009. 


Our audit did not cover how CIC assesses whether applicants are 
admissible to Canada or how the provinces and territories nominate 
candidates for selection. Nor did we examine the Canada Border 
Services Agency’s processing of work permit applications at points of 


entry into Canada. 


Immigration has played an important role in the economic, social, and 
cultural development of Canada throughout our history. Its role is just 
as important today, given our aging population and labour force.: 
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What we found 


Canada has an ongoing need for permanent workers with various skills 
and must compete with other countries to attract them. In addition, 
Canada has a need for various types of temporary workers to address 
short-term needs of the labour market, which vary from year to year 
and from region to region of the country. 


It is critical that the government’s programs to facilitate the entry of 
permanent and temporary workers be designed and delivered in a way 
to ensure that the right people are available at the right time to meet 
the needs of the Canadian labour market. The choices that are made 
now will affect the kind of society Canada has in the future. 


¢ Although CIC followed a sound decision-making process in 2008 to 
design the Canadian Experience Class (a category of skilled foreign 
workers and students with Canadian work experience), the 
Department has made other key decisions without properly assessing 
their costs and benefits, risks, and potential impacts on other 
programs and delivery mechanisms. Program changes in recent years 
have resulted in a significant shift in the types of workers being 
admitted permanently to Canada under the immigration program’s 
economic component. We saw little evidence that this shift is part of 
any well-defined strategy to best meet the needs of the Canadian 
labour market. 


¢ The inventory of applications in the Federal Skilled Worker category 
has almost doubled since our 2000 audit and, in December 2008, 
represented more than 620,000 people waiting an average of 
63 months for a decision on whether they would be admitted. 
Measures taken by CIC in 2008 to limit the number of new 
applications—for example, processing only those that meet new, 
more narrowly defined criteria—were not based on sufficient analysis 
of their potential effects. While it is too early to assess their full 
impact, trends in the number of applications received since the 
beginning of 2009 indicate that they might not have the desired 
effect, and CIC could be unable to process new applications within 
the 6 to 12 months it has forecast. Furthermore, CIC does not know 
and has not defined how much time it should take to clear the 
inventory of applications on hand when the measures were 
introduced. 


¢ CIC and HRSDC have not clearly defined their respective roles and 
responsibilities in assessing the genuineness of job offers and how 
that assessment is to be carried out. As a result, work permits could 
be issued to temporary foreign workers for employers or jobs that do 
not exist. In addition, there is no systematic follow-up by either 
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department to verify that in their previous and current employment 
of temporary foreign workers, employers have complied with the 
terms and conditions (such as wages and accommodations) under 
which the work permits were issued. This creates. risks to program 
integrity and could leave many foreign workers in a vulnerable 
position, particularly those who are physically or linguistically 
isolated from the general community or are unaware of their rights. 
Furthermore, weaknesses in the practices for issuing labour market 
opinions raise questions about the quality and consistency of 


decisions being made by HRSDC officers. 


¢ CIC has successfully introduced a number of initiatives and tools to 
address some of the inefficiencies we reported in 2000 in its 
processing of applications in missions overseas. However, efficiency 
gains will be seriously limited until an information technology system 
that has been under development for almost 10 years is implemented 
in missions abroad and CIC makes effective use of available 
technologies. In the meantime, employees in missions abroad are still 
buried in paperwork and spending a great deal of their time on 
clerical tasks. In addition, while the Department has developed a 
quality assurance framework that is available to all missions, 
immigration program managers are not required to use it or to report 
on quality assurance. Therefore, CIC still has little assurance that 
overall, decisions by visa officers are fair and consistent. 


The entities have responded. The entities agree with all of the 
recommendations. Their responses follow each recommendation 
throughout the chapter. 
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Introduction 


Overview of the Canadian immigration program 


2.1. Immigration plays an important role in the economic, social, 
and cultural development of Canada. According to Statistics Canada, 
immigration accounted for two thirds of Canada’s population growth 
in 2006. At the same time, the existing population is aging and 

the working population, a diminishing proportion of the total, could 
become too small to respond to the economic and labour market needs 
of the country. The federal government has determined 

that immigration is part of the solution, both now and in the future. 
However, in attracting economic immigrants, Canada must compete 
with many industrialized countries facing similar circumstances. 


2.2 Meeting these challenges requires legislation, regulations, 
and programs that can attract foreign workers to fill Canada’s needs 
without compromising the health and safety of its citizens. 


2.3 Mandate and authority. The Immigration and Refugee Protection 
Act (IRPA) that now governs immigration in Canada came into effect 
on 28 June 2002, replacing the Immigration Act of 1976. Among other 
objectives, the IRPA is intended to | 


* support the development of a strong and prosperous Canadian 
economy, in which the benefits of immigration are shared across 
all regions of Canada; 


* support, by means of consistent standards and prompt processing, 
the attainment of immigration goals established by the Government 
of Canada in consultation with the provinces and territories; 


¢ facilitate the entry of visitors, students, and temporary workers 
for purposes such as trade, commerce, tourism, international 
understanding and cultural, educational, and scientific activities; 


and 


* work in cooperation with the provinces and territories to 
secure better recognition of the foreign credentials of permanent 
residents and their more rapid integration into society. 


24 Citizenship and Immigration Canada (CIC) is a key player 
in the application of the IRPA. Under specific immigration programs, 
the Department 

© develops Canada’s policy on selection of applicants for permanent 


and temporary residency, 
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Mission—An office of the Government of 
Canada outside Canada. This includes an 


embassy or high commission, consulate general, 


and consulate. 


6 Chapter 2 


* sets the conditions under which they may enter and remain in 


Canada, 


* screens applicants to determine whether they present any risk to 
Canadians, 


* processes applications from foreign nationals for either permanent 
or temporary residency, and 


* assists with settling them in Canada and integrating them into 
society. 


2.5 Human Resources and Skills Development Canada (HRSDC) is 
responsible for the design, management, and delivery of those elements 
of temporary foreign worker programming that fall under its Minister’s 
responsibility, including the issuance of labour market opinions to 
employers and to CIC. It delivers services through its Service Canada 
offices in all regions of Canada. HRSDC is also involved in the federal 
efforts to facilitate the recognition of foreign credentials. 


2.6 $Anumber of other federal entities, including the Canada Border 
Services Agency (CBSA), are also involved in the delivery of the 
immigration program. 


2.7. Under the Constitution Act, immigration falls under the 
leadership of the federal government but responsibility is shared with 
the provinces and territories. Successful management of the 
immigration program therefore requires effective collaboration 
between the federal and the provincial and territorial governments. 
Under the IRPA, the Minister of Citizenship, Immigration and 
Multiculturalism has the authority to sign immigration agreements 
with the provinces and territories. 


2.8 The Canada-Quebec Accord, signed in 1991, grants Quebec the 
authority to set its own target levels for immigration each year and to 
select the immigrants who will go to the province. CIC also has 
agreements in place with most other provinces and territories for a 
provincial nominee program; these agreements give the province or 
territory the authority to pre-select and nominate individuals who fill a 
specific need in its labour market. However, CIC remains responsible 
for the final selection decision. 


2.9 In the 2008-09 fiscal year, CIC had an annual budget of 

$1.32 billion, of which $434.7 million was for operating expenditures; 
funding for grants and contributions accounted for the balance. The 
Department had about 3,600 employees (full-time equivalents), 
including about 260 posted overseas. Immigration services abroad are 
provided through 89 offices in Canadian missions where, in addition to 
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Canadian employees, more than 1,260 locally engaged staff work. 
In 2008, about 247,000 immigration visas, as well as 1,261,000 temporary 
resident visas, and work and study permits were issued. 


2.10 Planning of immigration levels. The Immigration and Refugee 
Protection Act (IRPA) requires the Minister of Citizenship, Immigration 
and Multiculturalism to table a report on immigration in Parliament 
before 1 November each year. This annual report gives the number of 
newcomers who were admitted to Canada during the previous year and 
announces Canada’s immigration plan and target levels for the 
following calendar year. These levels are set for permanent residents 
only. There is no limit established for temporary residents. 


2.11 The Department plans immigration levels in consultation with 
the provinces and territories, certain federal departments, and national 
and local organizations with a particular interest in immigration. The 
Department also considers factors such as Canada’s capacity to absorb 
new immigrants, trends in international migration, and its own 
Capacity to process immigrants’ applications. Based on its analysis of 
the information it has gathered, the Department recommends to the 
Minister the immigration levels to be set. The annual immigration 
levels are then approved by the government before they are 
announced in Parliament. 


2.12 Annual target levels have remained essentially the same in 
recent years—from a minimum of 220,000 in 2004 to a minimum 

of 240,000 in 2009, broken down into various classes (Exhibit 2.1). 
The targeted number of immigrants to be admitted includes both the 
principal applicants and their dependants. 


2.13 Selection of foreign workers. Foreign workers can come 
to Canada under four categories of the Economic class to become 
permanent residents (Exhibit 2.2). As permanent residents, they 
are authorized to live and work in Canada as long as they retain 
their status. 


2.14 Foreign workers can also come to Canada to work on a 
temporary basis. Temporary foreign workers have access to four streams 


(Exhibit 2.3): 
¢ Skilled workers, 
* Occupations requiring lower levels of formal training (pilot project), 
¢ Live-in caregivers, and 


¢ Seasonal agricultural workers. 
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These foreign workers must obtain a work permit, unless exempted 
under the IRPA. Usually, a work permit is valid only for a specified job 
and length of time. In 2008; almost 370,000 work permits were issued. 


Exhibit 2.1 Annual target immigration levels have changed little since 2004 (thousands) 


Economic 132.0-148.0 132.5-148.0 126.0-143.0 141.0-158.0 139.0-154.0 140.3-156.6 


52.5-55.5 51.5-56.8 61.0-65.0 67.0-69.0 68.0-71.0 68.0-71.0 


Family 


Protected persons 29.4-32.8 30.8-33.8 32.8-40.3 25.9-30.8 26.0-31.8 23.6-27.2 


6.1.8.7 5.2-6.4 5.2-6.7 6.1-7.2 7.0-8.2 8.1-10.2 


220.0-245.0 | 220.0-245.0 | 225.0-255.0_ 240.0-265.0 | 240.0-265.0 


Economic class—Persons selected as permanent residents for their skills and ability to contribute to Canada’s economy. 


Others 


-240.0-265.0 


Family class—Close relatives of a Canadian citizen or permanent resident who may be sponsored to immigrate to Canada. Relatives are primarily spouses, 
partners, dependent children, parents and grandparents, but other family members may be eligible in certain circumstances. 


Protected persons—Persons abroad or in Canada who have been determined to be Convention refugees (United Nations Convention Relating to the Status of 
Refugees and Protocol to the Convention) or in need of protection as defined in the /mmigration and Refugee Protection Act. 


Others—Persons who, in exceptional circumstances, are granted permanent status even though they would ordinarily not qualify. For example, cases where there 
are strong humanitarian and compassionate considerations. 


Source: Citizenship and Immigration Canada reports on plans and priorities and annual reports to Parliament on immigration. 


Exhibit 2.2 Categories of the Economic class under which foreign workers may apply for permanent residency* 


Category Description - os 


Federal Skilled Worker (FSW) The FSW category selects immigrants for their skills. This category stresses education, 
English or French language abilities, and work experience in some occupations included 
in categories O (managerial), A (professional), and B (technical and skilled trades) of the 
National Occupational Classification (NOC) system. Applications are assessed against a 
point system.** 


Canadian Experience Class (CEC) The CEC selects skilled temporary workers with Canadian work experience and 
international students with Canadian degrees and Canadian work experience; they may 
apply for permanent residency without leaving the country. 


Quebec Skilled Worker (QSW) The Canada-Quebec Accord grants Quebec the authority to set annual immigration 
targets and the responsibility for selecting skilled worker immigrants. (Our audit did not 
include this category.) 


Provincial Nominee Program (PNP) The PNP allows provincial and territorial governments to pre-select and nominate 
applicants based on their own selection criteria. However, Citizenship and Immigration 
Canada is responsible for the final selection decision. 


* The Economic class also includes the business categories of investors, entrepreneurs, and self-employed applicants, none of which were included in our audit. 
Live-in caregivers may also become permanent residents but they are first admitted to Canada as temporary foreign workers. 


**For information on the point system, visit Citizenship and Immigration Canada’s website at www.cic.gc.ca. 


Source: Citizenship and Immigration Canada annual reports to Parliament on immigration, and /mmigration and Refugee Protection Regulations 
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Exhibit 2.3 Temporary foreign worker streams 


Skilled workers 
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2.15 Operating environment. Our last audit of the immigration 
program’s economic component was published in our April 2000 
Report. Since then, there have been major changes in the delivery of 
immigration programs. In particular, the IRPA was enacted in 2002; 
the Canada Border Services Agency (CBSA) was created in 2003 and 
took over all enforcement and control activities from CIC. Both of 
these events generated implementation challenges. 


2.16 In addition, Canada’s economic growth in recent years 
substantially increased the demand for temporary foreign workers and 
the number of temporary foreign worker applications. 


2.17 In 2006, the government recognized the need to more closely 
align immigration policies with labour market needs while addressing 
local shortages, through greater use of the Provincial Nominee 
Program and by encouraging skilled temporary foreign workers with 
Canadian work experience to apply for permanent residency. Finally, 
in 2008 the government committed to $124 million over a five-year 
period to modernize the immigration system and make it more 
competitive and responsive. 


2.18 CIC’s performance, among others, is measured by its success 
in achieving the annual target immigration levels. As shown in 
Exhibit 2.1, these target levels have increased only slightly since 2004. 


Description ; Te Nap che ee 


Employers may wish to hire foreign workers to work temporarily in occupations or sectors 
of the economy that require workers with a high level of formal education or training 
included in categories O (managerial), A (professional), and B (technical and skilled 
trades) of the National Occupational Classification (NOC) system. These usually require a 
college or university degree. 


SS |_| i sSsSS5005905050500 TE 


Occupations requiring lower levels of 
formal training (pilot project) 


This stream allows employers to hire temporary workers for lower-skilled occupations 
included in categories C and D of the NOC system, which typically require at most a 
high-school diploma or a maximum two years of job-specific training. This is a pilot 
program that was introduced in July 2002. 


eee”? 02  s ee LL eke see _— 


Live-in caregivers 


Live-in caregivers are admitted to Canada on a temporary basis and may apply for 
permanent residency from within Canada after being employed as a caregiver for two out 
of three years. 


Seasonal agricultural workers 


The seasonal agricultural worker stream allows for the organized entry of temporary 
workers from Mexico and a number of Caribbean countries into Canada to meet the 
seasonal needs of agricultural producers when Canadian workers are not readily 


available. 


Source: Citizenship and Immigration Canada annual reports to Parliament on immigration, the Temporary Foreign Worker Program Integrated Results-Based 
Management and Accountability Framework, and the Risk-Based Audit Framework 
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Strategic planning and 


programming 


CIC’s resource base in missions abroad, which handle most applications 
for permanent and temporary residency, has also remained relatively 
constant during a period when temporary foreign worker applications 
increased significantly. Nonetheless, CIC has been able to achieve the 
annual immigration target levels with the sole exception of 2007, 
when 236,758 immigrants were admitted, slightly below 

the 240,000 minimum target. 


Focus of the audit 


2.19 We examined how CIC and HRSDC plan for and manage 
programs designed to facilitate the entry of permanent and temporary 
workers into Canada. We also examined how applications are 
processed and what is being done to facilitate the recognition of 
foreign credentials in Canada. We did not examine how CIC 
determines applicants’ admissibility in terms of health, safety, and 
security. Nor did we audit the Canada Border Services Agency’s 
processing of work permit applications at Canadian points of entry or 
how the provinces and territories nominate candidates for selection. 


2.20 We carried out our audit work at CIC offices in Canada and in 
eight missions overseas, and in HRSDC offices in Canada. 


2.21 More details on the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


2.22 Canada has an ongoing need for permanent and temporary 
workers with various skills and must compete with other countries to 
attract them. It is therefore critical that the government’s programs to 
facilitate the entry of these workers to Canada be designed in such a 
way that the right people are available at the right time to meet the 
needs of the Canadian labour market. 


2.23 As outlined further in this chapter, a number of important 
changes have been made to existing programs, and new programs have 
been introduced in recent years. We examined whether these 
modifications were based on a clear vision of the type and number of 
foreign workers to be admitted to Canada under each program in order 
to meet the needs of the Canadian labour market. 
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What each program should contribute to meeting Canadian labour market needs 
has not been defined 


2.24 Annual target levels for the Economic class as a whole have 
changed little in the last six years, ranging from a minimum of 132,000 
in 2004 to 140,300 in 2009. However, there has been a significant shift 
in the number of immigrants Canada will admit under each category 


within that class (Exhibit 2.4). 


2.25 For example, between 2004 and 2009, increased economic activity 
in a number of provinces and lengthy delays in processing Federal Skilled 
Worker applications resulted in an increase of almost 471 percent in the 
minimum target levels for the Provincial Nominee Program (PNP) 

category. During the same period, minimum target levels for the Federal 


Skilled Worker (FSW) category dropped 31 percent. 


2.26 We also noted that the immigration targets for the Economic 
class are influenced more and more by the movement of some 
temporary workers and international students who, after two years 

in Canada, have access to permanent residency through the Live-in 
Caregiver category and the recently created Canadian Experience 
Class category. There are no limits on the number of temporary foreign 
workers and international students being admitted each year. 
Consequently, target levels for the Live-in Caregiver and the Canadian 
Experience Class categories need to be adjusted each year to take into 
account the expected number of permanent residency applications to 


Exhibit 2.4 A significant shift has occurred in recent years in the number of immigrants that Canada planned to accept under each category of 


the Economic class 


Planned target levels, including dependants (thousands) 


Category 2004 2005 2006 2007 ‘i 2008 2009 
Federal Skilled Worker 98.2-112.7 | 86.0-97.5 | 78.5-89.0 | 90.0-100.5 | 67.0-70.0 | 68.2-72.0 
| | | 

Quebec Skilled Worker 913-298 | 265-27.0 | 26.5270 | 26.0-27.5 | 250-280 | 28.1-29.1 
Federal/Quebec Business 6.0-6.0 9.5-10.5 9.0-11.0 go-to: |) {recs |) Mbi20 
Provincial Nominee 3.53.5 | 80-100 9.0-11.0 | 13.0-14.0 | 20.0-22.0 | 20.0-26.0 
Program : 

| | | | | d 
Live-in Caregiver 3.0-3.0 2.5-3.0 3.0-5.0 3.0-5.0 6.0-9.0 8.0-10.0 
Canadian Experience N/A N/A N/A N/A 10.0-12.0 5.0-7.5 
Class* 
Total Economic class 132.0-148.0 | 132.5-148.0 | 126.0-143.0 | 141.0-158.0  139.0-154.0 | 140,3-156.6 


*New category introduced in 2008 
Source: Citizenship and Immigration Canada’s annual reports to Parliament on immigration and reports on plans and priorities 
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be received from these temporary residents. For example, the 
significant increase in the number of work permits issued for live-in 
caregivers in recent years has made it necessary to almost triple the - 
minimum target level for this category since 2004. 


2.21 Citizenship and Immigration Canada (CIC) has made a number 
of projections regarding the target levels that could be required in 
future years for each category of the Economic class, if no changes are 
made to current programs or to the target for the Economic class as a 
whole. These projections indicate a significant change in the types of 
foreign workers that could be admitted to Canada in the next 

three years (Exhibit 2.5). For example, the Provincial Nominee 
Program category could soon become the largest source of economic 
immigrants in Canada, while the target level for the Federal Skilled 
Worker category could be as low as 18,000 in 2012, down from a 
minimum target level of 68,200 in 2009. 


2.28. Such change could have a significant impact on the ability of the 
immigration program to meet Canada’s labour needs. Although the 
Immigration and Refugee Protection Act (IRPA) requires that 
immigration levels be set on an annual basis, in our view, it would be 
important for CIC to have a clear vision of how many immigrants 


Exhibit 2.5 CIC projections indicate a significant change in the immigration target levels of each 
category within the Economic class 


Immigration target levels (thousands) 


~ 2009 ; 

(minimum of the 2012 Average : 
Category ts | range) (projected) Variation +/- % — 
Federal Skilled Worker 68.2 18.0 -13.6% 
Quebec Skilled Worker 28.1 28.0 -0.4% 
Provincial Nominee 20.0 40.0 100.0% 
Program 
Federal/Quebec WA) Ae 0.0% 
Business 
Live-in Caregiver 8.0 10.0 25.0% 
Canadian Experience 5.0 26.3 = ~A26,0% 
Class 
Total 140.3 133352. -5.0% 

Re SAWS be apes tar eR pe GC eae Se a ra eee ees roe eae ni | 


Source: Citizenship and Immigration Canada’s (CIC) 2008 Annual Report to Parliament on Immigration 
and the Office of the Auditor General’s analysis on projections 
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should be selected under each category over a multi-year planning 
horizon, for example three years. The Department has yet to 
determine this. 


2.29 A longer-term vision to the setting of target levels would 
bring many benefits. For example, any program changes required 
to influence the mix among various categories could be made on a 
more timely basis. We also noted that some provinces are planning 
their immigration levels on a multi-year basis. Not doing so at the 
federal level could hinder CIC’s ability to meet provincial/territorial 
expectations regarding the number of provincial nominees that 
could be admitted to Canada. Furthermore, communicating 
information on planned target levels on a multi-year basis 

could generate informed discussions among parliamentarians 

and various stakeholders on the overall direction of the 
immigration program. 


2.30 In its 2004 Annual Report to Parliament on Immigration, 
CIC committed to developing a national immigration framework in 
order to improve coordination with the provinces and territories 
and other stakeholders and ensure a flexible immigration program 
that meets Canada’s objectives. The Department reiterated in its 
2005—06 Report on Plans and Priorities that it would “work with 
partners to build upon the foundation created in recent years and, 
beginning in 2005-06... develop a new immigration framework for 
the future.” 


2.31 The Department’s intent for the national framework it 

proposed was to provide a strategic roadmap for the future of the 
immigration program. This roadmap would facilitate the assessment 

of the potential impacts of proposed changes in immigration legislation, 
policies, and programs. In addition, it would enable the immigration 
program to respond more effectively to future challenges and address the 
challenges of the current delivery system, including rising inventories 
and delays in processing applications. 


2.32. Although we found that CIC has put in place an elaborate 
structure to coordinate immigration with the provinces and territories, 
including bilateral and provincial nominee agreements and regular 
federal-provincial/territorial meetings, it has not yet developed the 


strategic roadmap it was contemplating. 


2.33 In our view, such a strategic roadmap for the future of the 
immigration program would help to set a clear vision of what each 
program is expected to contribute toward meeting the objectives for 
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the Economic class and to ensure that CIC’s program mix is optimal 
and closely aligned with that vision. 


2.34 Recommendation. To strengthen its strategic planning, 
Citizenship and Immigration Canada should 


* develop, in consultation with provinces and territories, a strategic 
roadmap for the future of the immigration program including a 
clear vision of what each category is expected to contribute, on a 
multi-year basis, toward achieving the objectives of the Economic 
class; and 


¢ take the necessary steps to ensure that its programming is optimal 
and closely aligned with that vision. 


The Department’s response. Agreed. Work is already underway that 
will contribute to the vision and future of the immigration program. 
A notable example is the Department’s work on the broader 
modernization of the immigration program, including measures that 
have policy, operational, and service improvement and technological 
components. In addition, the evaluation of the Federal Skilled Worker 
program and work on regulatory changes to the Temporary Foreign 
Worker Program to address concerns around program integrity are 
two more current areas of activity that support the modernization of 
the immigration system. Taken together, this work will lead to the 
development of an immigration roadmap, which will be developed in 
consultation with our provincial and territorial partners over the next 
two years. 


In terms of the balance between categories in the Economic class, 
Citizenship and Immigration Canada launched a process in 
November 2008 with provinces and territories to develop a common 
approach for understanding how various programs under the 
Economic class, such as Federal Skilled Worker and Provincial 
Nominee contribute to meeting the economic objectives for 
immigration, such as responding to labour-market needs. Part of this 
effort involves achieving an appropriate balance among programs. 


Evaluations of the programs we audited are dated or have not been carried out 


2.35 We expected that programming decisions would be supported 
by timely and accurate information on the performance of existing 
programs. In Chapter | of this Report, Evaluating the Effectiveness 
of Programs, we note, “The government views evaluation as the 
primary source of neutral and systematic information on the ongoing 
relevance and performance of policies and programs.” 
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2.36 We found that evaluations of the programs we audited are either 


dated or have not been carried out. For example, CIC last evaluated the 
Federal Skilled Worker program in 1998, despite the significant changes 
made to this program following the introduction of the IRPA in 2002. 
Furthermore, CIC has never evaluated the Provincial Nominee and the 
Live-in Caregiver programs. The Department’s draft evaluation plan 
provides for an evaluation of the Federal Skilled Worker program in 

the 2009-10 fiscal year as well as evaluations of the Provincial Nominee 
Program and the Canadian Experience Class in the 2010-11 fiscal year. 
CIC informed us that an assessment of the Live-in Caregiver program 
will be included as part of a joint CIC/HRSDC evaluation of the 
Temporary Foreign Worker Program (TFWP). CIC also informed us that 
regulatory and policy obligations to evaluate grant and contribution 
programs on a priority basis delayed the evaluation of its other programs 
including those covered by our audit. — 


2.37 We also found a lack of timely program evaluation for the TFWP. 
For example, in 2002, Human Resources and Skills Development 
Canada (HRSDC) committed to the government to develop an 
evaluation framework for this program. In 2007, CIC and HRSDC 
further committed to conduct a formative evaluation in the 2009-10 
fiscal year and a summative evaluation in the 2011-12 fiscal year. An 
evaluation strategy for this program was finalized in August 2008. In 
April 2009, HRSDC informed the Treasury Board of Canada 
Secretariat that each of these evaluations will be postponed one year to 
allow for a more complete assessment of the program, taking into 
account recent and upcoming changes. 


2.38 Consequently, not only are CIC and HRSDC unable to 
determine the extent to which these programs are meeting their 
objectives and achieving their expected outcomes, they are missing 
valuable information that could be used when making changes to 
programs or designing new programs. 


2.39 Recommendation. Citizenship and Immigration Canada and 
Human Resources and Skills Development Canada should carry 

out evaluations of permanent and temporary foreign worker programs 
according to their approved evaluation plans. 


The departments’ response. Agreed. The departments have initiated 
work on a joint, comprehensive evaluation of the Temporary Foreign 
Worker Program that will provide results in 2010-11. CIC is currently 
carrying out a comprehensive evaluation of the Federal Skilled Worker 
program; other programs in the permanent stream will be evaluated 


according to the CIC evaluation plan. 
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Federal Skilled Worker category 
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The inventory of Federal Skilled Worker applications almost doubled since 2000 


2.40 One of the great challenges facing Citizenship and Immigration 
Canada (CIC) today is managing the large inventory of applications 
under the Federal Skilled Worker (FSW) category. Until recently, 

CIC had no means to control the number of applications submitted 
and it was required by the Immigration and Refugee Protection Act (IRPA) 
to process them all to a final decision. When the number of applications 
received in a year exceeds what is required to meet the target level for 
the category, applications can sit in an inventory for long periods before 
prospective immigrants receive a decision. In such cases, Canada risks 
losing good candidates to other countries that make more timely 
decisions. 


2.41 In addition, long delays can hamper efficiency, as information 
submitted by applicants needs to be updated and numerous requests ~ 
about the status of an application must be dealt with. Long delays also 
create uncertainty for applicants and prevent a timely response to the 
needs of the labour market. 


2.42 Since our last audit, the number of applicants awaiting a decision 
has almost doubled (Exhibit 2.6). At 31 December 1999, about 
330,000 people—skilled workers and their families—were waiting to 
have their applications processed; the average waiting time was around 


25 months. At 31 December 2008, more than 620,000 people in this 


category were waiting and the average processing time was 63 months. 


Exhibit 2.6 The inventory of applications of federal skilled workers awaiting a decision has increased 
significantly since the end of 1999 


Persons 

700,000 
aida / seers 
500,000 oe: 

400,000 aoe 

300,000 
200,000 | 

100,000 


1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 


Years 


*The Office of the Auditor General has estimated the inventory from 1999 to 2003 based on data obtained 
from Citizenship and Immigration Canada (CIC). 


Source: CIC International Region statistical reports as of 31 December of each year 


Report of the Auditor General of Canada—Fall 2009 


SELECTING FOREIGN WORKERS UNDER THE IMMIGRATION PROGRAM 


2.43 The size of the inventory and the average processing times vary 
significantly from mission to mission. At the end of December 2008, 10 
missions accounted for 81 percent of the inventory; processing times 
in these missions ranged from 25 months in Buffalo to 87 months 


in Accra (Exhibit 2.7). 


Exhibit 2.7 Ten missions account for 81 percent of the inventory of federal skilled workers waiting for 
their applications to be processed 


ns | a) | : Average aricasine ana 
. Missions 7 Inventory (people) - (months) 

New Delhi 133,406 US 
Londan 85,394 et 
Manila 69,866 66 
Islamabad 47,387 ‘3 
Damascus 44 280 76 
Hong Kong ( 35,701 34 
Singapore 28,378 50 

Buffalo 24,410 | 25 
Accra PETES) 87 
Beijing 16,021 45 
Subtotal (81%) oe 504,022 61; 
All missions overseas (100%) | 623075. «| ”:sC«OS 
*Average number of months required to complete 80 percent of cases. 
Source: Citizenship and Immigration Canada International Region statistical reports as 


of 31 December 2008 


2.44 Several factors have contributed to the growth of the inventory. 
The new Immigration and Refugee Protection Act had a significant 
impact on the inventory of applications for the Federal Skilled Worker 
category. As Exhibit 2.6 shows, in 2000 and 2001, following the 
announcement that legislative changes were planned, there was a 
significant increase in the number of applications submitted before 
the new legislation came into effect. This was followed in the next 
two years by a drastic decline in applications, when it became apparent 
that the new selection criteria were more stringent. 


2.45 During the same period, perceived unfairness in the processing 
of applications caught in the transition between acts led to litigation 
by some prospective immigrants, resulting in a court injunction 
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For a list of the 38 eligible occupations, visit 
Citizenship and Immigration Canada’s 
website at www.cic.gc.ca. 


against CIC. While the injunction remained in effect—from June 2003 
to April 2005—no applications received prior to the introduction of 
IRPA could be rejected on the basis that they did not meet the new 
criteria set in the Act. These applications could not be finalized and 
remained in the inventory. In addition, the Department became 
concerned that given the stéep decline in new applications, it would not 
be able to meet target levels in 2004 and 2005. Consequently, in 
September 2003, in order to attract more applicants, the Minister of 
Citizenship and Immigration announced that the passing’ mark for 
selection under this category had been reduced. As a result, the number 
of applications received after 2003 increased, adding to the inventory. 
The significant reduction in the target levels for the FSW category in 
recent years was also a key factor in the growth of the inventory. 


Measures have been taken to reduce the size of the inventory 


2.46 We examined whether the Department ensures that 
applications are processed within what it considers to be a reasonable 
period of time. 


2.471 We found that since 2004, CIC has considered various strategies 
to manage its inventory and reduce the number of applications 
received. For example, it considered seeking regulatory changes to 
allow it to impose limits on the intake of new applications. In 2006 

it also considered increasing the passing mark for selection under 

the Federal Skilled Worker category. In 2008, it finally proposed 
amendments to the IRPA to support a new inventory 

management strategy. 


2.48 In June 2008, the IRPA was amended, removing the 
requirement that all completed applications received be processed to a 
final decision. The amendments also authorized the Minister to give 
instructions for processing applications, including establishing the 
number to be processed in any given year and the order in which they 
are to be processed. 


2.49 The first ministerial instructions were issued 

on 28 November 2008; they established eligibility criteria for all FSW 
applications received after 26 February 2008, the date on which the 
intent to amend the IRPA was announced. To be eligible for 
processing, applicants must have one year of experience in one or more 
of 38 occupations identified by CIC; or have been in Canada legally 
for one year or longer as a temporary foreign worker or an international 
student; or have an offer of arranged employment. The main objectives 
of these eligibility criteria were to reduce the number of new 


18 Chapter 2 


Report of the Auditor General of Canada—Fall 2009 


SELECTING FOREIGN WORKERS UNDER THE IMMIGRATION PROGRAM 


applications and their processing times, and to ensure a better link 
between the selection of skilled workers and the needs of the labour 
market. Previously, all occupations under the National Occupational 
Classification 0, A, or B (351 occupations) were deemed eligible 

for processing. 


2.50 The Department expected that with significantly fewer 
occupations now eligible, the number of new applications would drop, 
and not all of them would be found eligible for processing. This would 
allow it to process all new applications as they came in, and it forecast 
that applicants would wait only 6 to 12 months for a decision. 
Operational instructions clearly indicate that eligible applications 
received on or after 27 February 2008 are to be processed as soon as 
possible and be given priority over older applications. 


2.51 The Department also expected that with fewer new applications 
to process, it could reduce the inventory of older applications by 

33 percent over five years. This was based on the assumption that at 
least 70 percent of annual target levels set for the FSW category would 
be met by processing applications already in the inventory. 


2.52 In the 2008-09 fiscal year, the Department also took other 
administrative measures to reduce the inventory—for example, 
contacting applicants already in the inventory to offer a refund of their 
processing fee if they decided to withdraw their application rather than 
continue waiting for a decision; and coding all applications received in 
its computerized system so that provinces could have the opportunity 
to contact applicants who could qualify under their provincial 
nominee programs. Additional resources were also allocated to help 
missions with larger inventories. However, these measures had little 
success in reducing the inventory of applications received prior 


to 27 February 2008. 


The inventory reduction strategy is not based on sufficient analysis 


2.53 CIC could not provide us with evidence of sufficient analysis to 
support the choice of the criteria set out in the ministerial instructions 
for assessing the eligibility of a FSW application. For example, it could 
not show us how it settled on reducing the number of admissible 
occupations as a criterion for eligibility, or any analysis of the extent to 
which this criterion would help reduce the number of new 
applications. Nor did it provide us with any analysis of its associated 
risks and its potential impact on the delivery of the program. 
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2.54 In addition, the Department could not show us any analysis for 
selecting as an eligibility criterion the applicant’s legal presence in 
Canada for one year or longer as a temporary foreign worker or an 
international student. 


2.55 Finally, the implementation of the ministerial instructions 
created some challenges for CIC. Between 27 February 2008 and 28 
November 2008, applicants continued to apply without knowing that 
the list of eligible occupations would be reduced from 351 to 38. More 
than 59,000 applications were received during that period. The 
assessment of these applications started in December 2008, and CIC 
informed us that it will be completed by the end of March 2010. The 
Department expects that many of these applications will be found 
ineligible. Since processing fees are collected when the application is 
submitted, refunds to these applicants all over the world have to be 
made. CIC has estimated that it will have processed almost 

45,000 refunds by the time this process is completed, which is now 
expected by the summer of 2010. 


2.56 Recommendation. In developing future ministerial instructions, 
Citizenship and Immigration Canada should ensure that selected 
strategies are supported by strong policy, program, and operational 
analysis. 


The Department’s response. Agreed. The Department agrees with 
the importance placed on strong policy analysis in the issuance of 
future ministerial instructions. 


To this end, the Department is working on several initiatives, including 
ensuring that diagnostic and analytical tools are in place for the 
monitoring and evaluation of the current ministerial instructions, 
improved analytical capacity around labour market information, and 
the exploration of options for expert advice to be regularly available to 
Citizenship and Immigration Canada regarding Canadian labour 
markets and current and projected occupational shortages. 


Trends in the number of eligible applications need to be monitored very closely 


2.57 It is too early to assess the full impact of the new eligibility 
criteria and to determine whether and to what extent they will reduce 
the number of new applications to be processed and their processing 
times. By the end of June 2009, however, the Department was 
experiencing only a slight decrease in the number of FSW applications 
received monthly compared with previous years, and close to 

80 percent of them appeared eligible to be processed. 
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2.58 If the eligibility criteria do not serve their intended purpose of 
lowering the number of new applications to be processed, CIC will 
need to react quickly and consider alternative strategies to achieve this 
objective. Failure to do so could result in the creation of a new 
inventory of applications submitted since the ministerial instructions 


came into effect—and the Department would be unable to process 


new applications within the 6 to 12 months it has forecast. 


2.59 In 2008, CIC estimated, based on a number of assumptions and 
the information available at the time, that the inventory of 
applications on hand at the end of February 2008 might not be 
eliminated for another 8 to 25 years. At the time of our audit, CIC had 
not revised its estimates to take into account information currently 
available and its experience to date with the new eligibility criteria. 
The Department was therefore unable to determine when this 
inventory will likely be eliminated nor has it defined what would be a 
reasonable timeframe to do so. CIC’s ability to reduce this inventory 
could be significantly impaired if the number of new applications to be 
processed is not reduced. The situation will be compounded if the 
target levels for the FSW category continue to be reduced. 


2.60 At the end of February 2008, there were 635,233 people 
(including dependants) in the old inventory of FSW applications. As 
of 30 June 2009, this inventory has been reduced to 451,948 people. 
However, such reduction was possible only because very few of the 
applications received since the ministerial instructions came into effect 
have been processed so far. Target levels for 2008 were met by 
processing only old applications. In future years, CIC will give priority 
to new applications and might process only a small percentage of old 
applications. 


2.61 As of 30 June 2009, the total inventory including both old 
and new applications amounted to 594,122 people, a reduction of 
6.5 percent since the effective date of the ministerial instructions. 
This reduction, however, is mainly explained by the high refusal 
rate of applications received between 27 February 2008 

and 28 November 2008 when eligibility criteria were unknown 


to applicants. 


2.62 Recommendation. Citizenship and Immigration Canada should 
closely monitor the impact of the new eligibility criteria on the number 
of applications to be processed under the Federal Skilled Worker 
category and on the effectiveness of its inventory reduction strategy. 
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The Department’s response. Agreed. The Department has 
introduced measures to ensure timely and comprehensive monitoring 
of the impact of the ministerial instructions. First, using existing data 
sources, the Department regularly tracks and analyzes data relating to, 
among other things, application intake rates, acceptance rates, and 
outcomes at key decision points throughout the processing continuum 
(i.e., from the Centralized Intake Office to processing and visa issuance 
at overseas missions). Second, the Department has introduced 
(effective June 2009) monthly reporting templates for overseas 
missions to capture more qualitative information on how the 
ministerial instructions are affecting the Federal Skilled Worker 
category. Taken together these initiatives will inform the need to revise 
ministerial instructions as appropriate. 


Strong structures and processes are needed to support the identification 
of priority occupations 


2.63 The significant reduction in the number of occupations eligible 
for the FSW category represents a return to an approach driven by 
occupations in demand, similar to the approach in place before the 
Immigration and Refugee Protection Act (IRPA) was enacted. Identifying 
occupations in demand is a key mechanism chosen by the government 
to ensure a better link between the needs of the labour market and the 
selection of federal skilled workers. In our view, it is important that 
strong structures and processes support this mechanism. 


2.64 In our 2000 Report, we identified some deficiencies in the 
structures and processes in place to define what the Department called 
at the time “occupations in demand in Canada.” We noted then that 
the list of occupations that was used to select potential immigrants had 
not been updated for seven years and therefore might not be a true 
reflection of the Canadian labour market’s needs. 


2.65 In this audit, we found that CIC held extensive consultations 
with various stakeholders, including HRSDC and provinces and 
territories, before the ministerial instructions were issued. However, 
CIC has not determined the process and mechanisms needed to ensure 
that the list of occupations in demand remains up-to-date and reflects 
the needs of the labour market. In comparison, we noted that other 
countries carry out regularly scheduled, expert-based reviews and 
updates of listed occupations in demand. 


2.66 Furthermore, the majority of occupations listed under the 
ministerial instructions are regulated professions. Successful applicants 
admitted to Canada for those occupations will need to have their 
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credentials recognized by appropriate provincial boards or professional 
associations. Difficulties in obtaining that recognition could limit an 
applicant's ability to find employment in the occupation for which he 
or she was allowed to apply for permanent residency. 


2.67 Whether there was a reasonable prospect of foreign credential 
recognition was taken into account in identifying priority occupations. 
However, given the importance of foreign credential recognition to the 
successful integration of immigrants into the labour market, sustained 
effort and coordination of ongoing federal initiatives will be required. 
(Some of the federal initiatives are described under “Recognition of 
foreign credentials” paragraphs 2.130 to 2.138). 


2.68 Recommendation. Citizenship and Immigration Canada, in 
consultation with key partners, should put in place structures and 
processes to ensure that the selected eligible occupations remain 
relevant to the changing needs of the Canadian labour market. 


The Department’s response. Agreed. Timely and accurate labour 
market information is an important component of ensuring that the 
ministerial instructions help meet the Government of Canada’s goal of 
improving the labour market responsiveness of the immigration 
program. Over the next two years, Citizenship and Immigration 
Canada will work with key partners to explore options for structures 
and processes to better ensure the instructions meet their objectives. 
Part of this work includes exploring options for an external advisory 
body, such as the U.K.’s Migration Advisory Committee. 


Limited analysis was carried out to support the creation of the Centralized Intake 
Office for Federal Skilled Worker applications 


2.69 In conjunction with the ministerial instructions issued in 
November 2008, the Department decided that the intake of Federal 
Skilled Worker applications would be centralized in Canada. 
Applicants are now instructed to send their application to the 
Centralized Intake Office (CIO) in Sydney, Nova Scotia. The 
Department informed us that the CIO was created as a pilot project to 
test assumptions surrounding CIC’s long-term vision for client service, 
including centralizing the front-end processes of file creation in 
Canada. An initial budget of $7.9 million over two years has been 
allocated for the CIO, with an additional $18 million over three years 
to be reassessed as information becomes available on its effectiveness. 


2.70 The CIO receives applications without supporting 
documentation, creates files and enters data, collects processing fees, 
and pre-screens applications to determine whether an application 
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appears to be eligible under ministerial instructions. The information 
provided by applicants is not validated at this stage. When an 
application does not meet the eligibility criteria, the CIO informs 
the applicant and refunds the fees. 


2.71 If an application appears to meet the eligibility criteria for 
processing, the file is transferred electronically to the appropriate 
mission abroad and the applicant is instructed to resubmit his/her 
application with all required supporting documentation directly to 
the mission. 


272 While part of a broader initiative to modernize the immigration 
system, the CIO is meant to provide for consistency in implementing 
ministerial instructions and to facilitate the management of all fees. It 
is expected to reduce the workload of missions, since they would not 
have to process ineligible applications or manage fees. 


2.73 In 2008, however, many applications to which the ministerial 
instructions would apply retroactively had been received in missions 
before the instructions were issued and before the CIO opened. 
Missions were therefore variously involved in managing these files. 
While some missions created files, collected fees, and completed 
preliminary eligibility assessments, others sent those files to the CIO in 
Sydney when it opened, for eligibility review or management of fees. 


2.14 Given the investment involved and the potential impact on 
CIC’s program delivery mechanisms, we expected that the decision 

to establish the CIO would be based on a sound analysis of its costs 
and benefits, risks, and potential impacts on program delivery. We 
found that although a number of options were considered, this was not 
the case. 


2.15 We also found that the Department encountered many problems 
in managing application fees when it centralized the intake of the FSW 
applications. The CIO accepts payments in Canadian currency only. 
However, a number of applicants did not have access to Canadian 
funds in their countries. In addition, the CIO was unable to reimburse 
applicants in eight countries where cheques issued by the Canadian 
government were not accepted. 


2.16 The Department has recognized that it will need to evaluate 
the CIO’s cost effectiveness in light of experience gained over the 
first two years of its operation, and that assumptions surrounding 
its long-term vision for client service—including centralizing the 
front-end processes of file creation in Canada—might need to 


be modified. 
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2.71 As of June 2009, results indicate that most files received in the 
CIO appear to be eligible for processing and are referred to missions 
abroad. This and the fact that CIC is considering electronic methods 
of payment for applicants could call into question the cost 
effectiveness and added value of this office, given its current mandate. 


2.18 Recommendation. Citizenship and Immigration Canada should 
evaluate, after the first two years of operations, the cost-effectiveness 
of the Centralized Intake Office and the extent to which it is meeting 
its objectives. 


The Department’s response. Agreed. The Department intends to 
continue to move on its modernization agenda as quickly as possible 
with the support of the roll-out of our Global Case Management 
System. Within our vision of moving some work back to Canada and 
working in a more virtual network, the Centralized Intake Office 
operations will be evaluated and a decision made on its 
cost-effectiveness and its support to meet CIC overarching objectives. 
The evaluation of the Centralized Intake Office is included in the 
Department’s Evaluation Plan in 2010/2011. 


2.19 The Provincial Nominee Program (PNP) enables provinces and 
territories, using their own selection criteria, to nominate candidates 
for admission as permanent residents to meet their specific labour 
market needs. Agreements with provinces and territories commit 
Citizenship and Immigration Canada (CIC) to process all provincial 
nominee applications as expeditiously as possible. 


A number of issues require attention 


2.80 Provincial nominees are not subject to the requirements of the 
point system applicable to the Federal Skilled Worker (FSW) category, 
nor does CIC impose any minimum selection threshold for these 
candidates. Provincial nominee programs have become highly diverse 
and complex over time, with selection criteria that vary substantially 
from one province to another. At the time of our audit, they included 
more than 50 different categories, each with its own selection 
approach and criteria. PNP agreements do not require provinces and 
territories to obtain CIC’s approval when they create new PNP 
categories; they are required only to inform CIC. 


281 In our view, given their increasing diversity and significant 
erowth in recent years, it would be time to take stock of these programs 
and formally evaluate them to assess the extent to which they are 
achieving their intended results. PNP agreements call for CIC to 
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negotiate evaluation plans with the provinces and territories. 

The purpose is to ensure that provinces and territories will perform 
sufficient analysis at appropriate intervals to produce data that can 
be used to inform discussions about any modification that might be 
required to these agreements. Currently, no evaluation plans have 
been negotiated with the provinces and territories. 


2.82 CIC officials informed us that an evaluation of the PNP from a 
national perspective is currently scheduled for the 2010-11 fiscal year. 
The Department is working with provinces and territories in an 
attempt to develop criteria that will be common to all evaluations, 
with resulting evaluation information to be shared with CIC on a 
regular basis. It will be important that the proposed national 
evaluation be comprehensive and include all provincial and territorial 
information to which the Department is entitled under the various 
PNP agreements. 


2.83 CIC officials also informed us that although PNP agreements 
require the provinces and territories to collect information on the 
retention of nominees within their respective jurisdictions, the 
information is either absent or incomplete and not always shared with 
the Department. The lack of information on the retention of nominees 
was raised in recent reports of three provincial auditors general in 
which one specifically noted that this represented non-compliance 
with the PNP agreement. 


2.84 All but one of the PNP agreements also state that CIC will 
consider a nomination certificate issued by a province or territory as a 
determination that it has conducted due diligence to ensure that the 
applicant has the ability and is likely to become economically established 
in that province or territory. However, we found that CIC has not 
defined what constitutes due diligence, both in terms of how new 
categories are established and how applications are processed by 
provinces and territories. Nor has it justified the basis on which it can 
assume that individual provinces or territories will exercise due 
diligence, or assessed the extent to which they have carried it out. The 
Department has also not determined whether provinces and territories 
have implemented mechanisms such as a quality assurance framework to 
ensure that their nomination decisions are consistent and compliant 
with the Immigration and Refugee Protection Act (IRPA) and its 
Regulations, and their PNP criteria. | 


2.85 Recommendation. Citizenship and Immigration Canada should 
work with provinces and territories to ensure that, for the Provincial 
Nominee Program, 
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* mechanisms are in place to collect and share appropriate 
information against agreed upon evaluation criteria; 


* evaluation plans are prepared and implemented to assess whether 
provincial nominee programs are meeting the objectives set out in 
the Immigration and Refugee Protection Act and Regulations; and 


* quality assurance mechanisms are in place to ensure that 
nomination decisions are consistent and compliant with the 
Immigration and Refugee Protection Act and Regulations and their 
respective Provincial Nominee Program criteria. 


The Department’s response. Agreed. The Department has developed 
an evaluation framework and logic model for an evaluation of the 
Provincial Nominee Program, which is in the Department’s Evaluation 
Plan for 2010/11. Common performance measures and indicators will 
be drawn from the longitudinal immigration database, which will 
provide information related to outcomes and retention for all 
provinces and territories. The Department remains open and 
committed to working with provinces and territories on any additional 
information sources and mechanisms they wish to have considered. 


The evaluation will assess performance of provincial nominee 
programs against the objectives of the Immigration and Refugee 
Protection Act and Regulations. 


Citizenship and Immigration Canada agrees with the need for quality 
assurance mechanisms and is open to working with provinces and 
territories to develop them. Should there be questions related to 
quality assurance, CIC can substitute its own evaluation of provincial 
and territorial nominations. 


Development of the Canadian Experience Class category is a good example of how 
programming decisions should be supported 


2.86 Studies ordered by Citizenship and Immigration Canada (CIC) 
in 2006 indicated that skilled workers with Canadian experience do 
better as permanent residents than those without it. Canadian 
experience before immigrating is mainly acquired through participating 
in the Temporary Foreign Worker Program or by working while 
studying in a Canadian institution. However, the CIC studies also 
indicated that only a small proportion of temporary foreign workers 
and international students apply for permanent resident status. The 
Department therefore believed there was a need for an effective bridge 
between temporary and permanent resident status. 


Report of the Auditor General of Canada—fall 2009 


Chapter 2 27 


SELECTING FOREIGN WORKERS UNDER THE IMMIGRATION PROGRAM 


The Temporary Foreign Worker 


Program 


2.87 Consequently, the Canadian Experience Class category was 
launched in September 2008. This category allows skilled temporary 
foreign workers with Canadian work experience and international 
students with Canadian degrees and Canadian work experience to 
apply for permanent residency based on specific criteria related to their 
work experience and their knowledge of Canada’s official languages. 
They do not have to leave the country to apply and do not have to 
meet the requirements of the point system applicable to the Federal 
Skilled Worker category. The Department estimated that the number 
of successful applicants in the Canadian Experience Class category will 
rise from a minimum of 5,000 in 2009 to more than 26,000 in 2012 
category. 


2.88 We found that the work carried out by CIC to design and 
implement the Canadian Experience Class category included detailed 
options analysis, assessments of potential impacts on other programs, 
and a detailed risk assessment. In our view, this is a good example of 
how programming decisions should be supported. 


2.89 ‘To address short-term regional or national shortages of skills or 
labour, Canada has a variety of programs designed to facilitate the 
entry of temporary foreign workers. These workers need to obtain a 
work permit from Citizenship and Immigration Canada (CIC), unless 
exempted under the Immigration and Refugee Protection Regulations 
(IRPR). Usually, a work permit is valid only for a specified job and 
length of time. There are no established limits on or target levels for 
the number of workers to be admitted under these programs. The 
number of applications processed each year depends mainly on the 
demand from employers. Human Resources and Skills Development 
Canada (HRSDC) and CIC share responsibility for the design and 


implementation of the Temporary Foreign Worker Program. 


2.90 The number of temporary work permit applications received 
abroad rose from 91,270 in 2002 to 204,783 in 2008, an increase of 
more than 124 percent. From 2007 to 2008 alone, the increase 

was 26 percent. This significant increase was due mainly to increased 
economic activity in various parts of the country and large projects 
needing temporary workers, such as oil sands development in Alberta 
and infrastructure development in British Columbia to support 

the 2010 Olympic and Paralympic Games. The long delays and the 
criteria used in the processing of applications under the Federal Skilled 
Worker (FSW) category also led employers to rely on temporary 
workers to address their immediate labour needs. 
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2.91 Before offering temporary employment to foreign workers, a 
prospective employer must apply to HRSDC for a labour market 
opinion (LMO) unless exempted by the IRPR. HRSDC issues the 
opinion to both the employer and CIC, which can then process the 
application for the work permit. 


2.92 Given the significant increase in the demand for temporary 
foreign workers, the government’s 2007 Budget provided close to 
$150 million in additional funding to CIC and HRSDC over five years, 
with $35.5 million annually thereafter, to improve the processing of 
applications. The intent was to reduce delays, respond more effectively 
to regional labour shortages, and improve the consistency and fairness 
of decision-making. Close to 80 percent of the funding went to 
HRSDC for processing applications for labour market opinions. 


2.93 Section 203(1) of the IRPR states, “On application. . . for a work 
permit made by a foreign national . . . an officer [of CIC] shall 
determine, on the basis of an opinion provided by the Department of 
Human Resources Development, if the job offer is genuine and if the 
employment of the foreign national is likely to have a neutral or 
positive effect on the labour market in Canada.” The labour market 
opinion issued by HRSDC through its Service Canada regional offices 
is consequently the key tool used by CIC officers to make that 
determination. CIC officers in missions abroad must also ensure that 
the applicant is qualified to perform the work and intends to leave 
Canada when the work permit expires. 


2.94. HRSDC provides functional direction, support, and guidance to 
its regional offices on how to deliver LMOs consistently, taking into 
account the particular nature of the local or regional labour market. 
The regional offices are responsible for processing LMO applications 
from employers and delivering program services in their region. 


2.95 The IRPR include six factors to be considered in assessing the 
likely impact that the proposed entry of a temporary foreign worker will 
have on the Canadian labour market. Among these factors are 
whether an employer who applies for an LMO can demonstrate that 
efforts were made to recruit or train Canadian citizens or permanent 
residents before resorting to hiring a temporary foreign worker, that the 
wages offered are consistent with the prevailing regional wage for the 
occupation, and that the working conditions for the temporary foreign 
worker meet generally accepted Canadian standards. However, the 
Regulations are silent on the factors to be considered in assessing the 


genuineness of a job offer. 
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Human Resources and Skills Development Canada’s practices do not ensure the 
quality and consistency of decisions when issuing labour market opinions 


2.96 We expected that HRSDC would process LMO applications 
efficiently and that it would have mechanisms in place to ensure that 
decisions were uniform and.in compliance with the Immigration and 
Refugee Protection Act and Regulations, and the Department’s own 
established procedures. 


2.97 We reviewed the process in place at HRSDC to issue labour 
market opinions, using a statistical sample of LMO applications 
processed between | October 2007 and 30 September 2008, along 
with the documentation provided to HRSDC by the employers. We 
also interviewed HRSDC employees at the Department’s headquarters 
and in four regional offices. 


2.98 We found that HRSDC has implemented a number of initiatives 
aimed at improving the administration of labour market opinions. For 
example, in Alberta and British Columbia it established a pilot project, 
the Expedited Labour Market Opinion (e-LMO), which significantly 
reduced the time it took officers to process LMO applications that were 
part of the pilot. Other initiatives include centralizing the processing of 
LMOs for live-in caregiver positions to shorten processing times and 
make more consistent decisions. Due to initiatives such as these and to 
staff increases, average processing times for LMO applications overall 
improved from 38 days in the 2007-08 fiscal year to 17 days in 

the 2008-09 fiscal year. 


2.99 However, we found that directives on how to assess whether 
employers meet some or all of the factors outlined in the Regulations 
are not clear or are incomplete; interpretations vary from one regional 
office to another and even within the same office. For example, 
directives on determining prevailing wages do not provide specific 
guidance and are not well understood by HRSDC officers. 
Furthermore, each regional office uses labour market information 
differently to assess and determine prevailing wages. Also, until 
January 2009, directives on how to assess whether employers have 
made reasonable efforts to advertise job offers to Canadian citizens or 
permanent residents prior to requesting temporary foreign workers 
were not clear and did not provide criteria to perform such 
assessments. During the course of our audit, HRSDC has issued 
revised instructions aimed at clarifying employer requirements and 
providing clear and consistent evaluation criteria. We were also told by 
HRSDC officers that formal training was limited to an introductory 
course on how to assess applications for labour market opinions; it did 
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not include reviews or updates of procedures. HRSDC officials 
informed us that from March 2009 to May 2009, using updated 
training material, national headquarters staff conducted training 
sessions in the regions. 


2.100 Many of the files we reviewed lacked adequate documentation to 
support the opinion. There is no formal quality assurance system to 
ensure that opinions are consistent and compliant with the Act and 
Regulations. Toward the end of our audit, HRSDC initiated a project 
to develop a national quality assurance framework aimed at ensuring 
greater consistency of decisions across Canada. 


2.101 Recommendation. Human Resources and Skills Development 


Canada should 


¢ provide clear directives, tools, and training to officers engaged in 
issuing labour market opinions; and 


* implement a quality assurance framework to ensure the quality 
and consistency of opinions across Canada. 


The Department’s response. Agreed. The Department has already 
taken steps to build on the program efficiencies achieved with 

Budget 2007 funding that will further enhance the effectiveness of 
processing labour market opinions. In this regard, the Department has 
put in place mechanisms to strengthen coordination with regional staff 
to enhance integrity, operations, and service delivery. This includes 
working with regions to develop new directives and clearer bulletins to 
improve national consistency in operations and decision-making. In 
March 2009, an electronic mailbox was created for regional staff to 
seek direction and exchange information on issues related to policy 
and operations. The Department continues to hold annual 
coordination meetings and monthly teleconferences between national 
headquarters and Service Canada regions to discuss operational and 
procedural issues. In July 2009, a new monthly teleconference was 
launched specific to integrity issues. 


In June 2009, a new team was created to develop and implement a 
quality assurance framework that will strengthen the responsiveness, 
consistency, speed, and quality of decision-making when issuing labour 
market opinions. This initiative will also help to improve overall 
program integrity. The team is working closely with headquarters and 
regional staff to ensure the framework is informed by an understanding 
of the operational realities faced in each region, and can therefore be 
an effective tool to support national consistency in decision-making. 
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The genuineness of job offers is not systematically verified 


2.102 We found that HRSDC and CIC have not clearly defined their 
respective roles and responsibilities for assessing the genuineness of job 
offers and how that assessment is to be carried out. Both departments’ 
operational manuals are silent on this matter. 


2.103 Such an assessment would include, for example, ensuring that 
the employer exists, can afford to pay the established wages, and that 
there is a real need for the worker. 


2.104 In some of the missions we visited, we found that CIC officers 
were not verifying the genuineness of job offers since they were under 
the impression that this was done by HRSDC when issuing an LMO. 
In other missions, CIC officers told us that this was a CIC responsibility 
but very difficult for them to perform, in light of their limited 
knowledge of the Canadian labour market’s regional particularities, 
their workload, and their lack of access to employers. 


2.105 Our review of the LMOs issued by HRSDC confirmed that they 
provide an opinion only on labour market effect and not on the 
genuineness of job offers. HRSDC officers informed us that, although 
they considered themselves better positioned than CIC officers to 
perform such assessments, they were under the impression they did not 
have the authority to do so. 


2.106 This lack of systematic assessment of the genuineness of job 
offers creates significant risks to the integrity of the Temporary Foreign 
Worker Program since work permits could be issued for employers or 
jobs that do not exist. 


2.107 Recommendation. Citizenship and Immigration Canada and 
Human Resources and Skills Development Canada should clarify their 
respective roles and responsibilities, and put mechanisms in place to 
ensure that the genuineness of job offers is systematically verified. 


The departments’ response. Agreed. The departments are working 
together to make improvements in these areas. The departments are 
also developing regulatory options to strengthen the integrity of the 
Temporary Foreign Worker Program including clarity around roles and 
responsibilities. Part of this work focuses on more structured 
assessments of the genuineness of both employers and jobs offered to 
temporary foreign workers. A new information-sharing agreement 
between CIC and HRSDC will enable a more active exchange of 
information pertinent to the assessment of the genuineness of job offers. 
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Additionally, new interdepartmental working groups and 
information-exchange mechanisms have been established to ensure 
better communication and coordination of program development and 
implementation. 


Concerns remain about the integrity of the program and the protection of temporary 
foreign workers ; 


2.108 Various studies and reports over the years have recognized that 
lower-skilled temporary foreign workers entering Canada may be 
vulnerable to exploitation or poor working conditions, usually because 
of their economic conditions, linguistic isolation, and limited 
understanding of their rights. 


2.109 For example, the Live-in Caregiver program permits applicants 
to be selected for permanent residency after being employed as a 
caregiver for two out of three years. There is a risk that live-in 
caregivers may tolerate abuse, poor working conditions, and poor 


accommodations so as not to lose the opportunity to become 


permanent residents. The program’s requirement that the caregiver 
reside in the employer’s home can put them particularly at risk. 

A number of CIC internal reports, some dating back as far as 1994, 
raised serious concerns about abuse of this program by employers and 
immigration consultants, as well as risks to individuals. 


2.110 Temporary foreign workers hired through the pilot project for 
occupations requiring lower levels of formal training may also be at risk 
of similar abuse and poor working conditions. Concerns over this 
vulnerability have grown with the surge in labour market opinion 
applications for this pilot project, which went from 12,627 in 2006 to 
68,568 in 2008. Some measures were taken to mitigate these risks 
when the pilot was launched in 2002, such as requiring an employment 
contract, employer-paid transportation, and health care. 


2.111 We found, however, that with the exception of e-LMOs, there 
has been no systematic follow-up by either CIC or HRSDC to verify 
that employers are complying with the terms and conditions under 
which the LMO application was approved, such as wages to be paid 
and accommodations to be provided. The e-LMO is a pilot project in 
which employers consent to compliance reviews in exchange for an 
expedited process. Employers who participate in the e-LMO pilot 
project are required to demonstrate adherence to wages, standards for 
working conditions, recruitment efforts, and other requirements. 
Employers that do not wish to participate in the pilot and to undergo a 
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compliance review have the option of applying for an LMO through 
the regular process. 


2.112 This lack of follow-up on job offers can have implications not 
only for the integrity of the programs but also for the well-being of 
foreign workers. CIC and HRSDC officials told us that neither the 
IRPA nor the Regulations give them authority to conduct compliance 
reviews of employers who have not consented. We were informed by 
both departments that regulatory modifications aimed at resolving 
some of these issues are currently being considered. 


2.113 The total number of applications received in missions overseas 
for the Live-in Caregiver program went from 6,178 in 2002 to 20,799 
in 2008, an increase of more than 236 percent. Given such growth and 
the inherent risks in this program, it is difficult to understand why it 
has never been formally evaluated by CIC since 1992, when it replaced 
the Foreign Domestic Movement Program. 


2.114 Furthermore, we noted that the pilot project for occupations 
requiring lower levels of formal training was launched with limited 
analysis of risks and without any formal goal, objectives, or basis on 
which to evaluate its success, nor has it been formally evaluated since 
then. It has been a pilot for seven years. Combined with live-in 
caregivers, temporary foreign workers under this pilot project now 
account for more than half of all temporary foreign workers in Canada. 


2.115 We found that immigration program managers and officers we 
interviewed in missions abroad were concerned about the level of 
misrepresentation or fraud from temporary foreign workers, employers, 
and their representatives in relation to work permit applications. 


HRSDC officers told us they have the same concerns about LMO 


applications from employers and their representatives. 


2.116 We noted that the IRPA does not provide any mechanism to 
impose administrative sanctions on employers or their representatives 
for compromising the integrity of the Temporary Foreign Worker 
Program. For example, an employer may submit a new LMO 
application even after misrepresenting a previous application. In our 
view, effective administrative sanctions would have a deterrent effect 
and could reduce the number of fraudulent applications. 


2.117 Recommendation. Citizenship and Immigration Canada (CIC) 
and Human Resources and Skills Development Canada (HRSDC) 
should implement mechanisms that would better enable them to 
ensure the integrity of the Temporary Foreign Worker Program and the 
protection of individuals. 


Report of the Auditor General of Canada—Fall 2009 


Processing of applications 


SELECTING FOREIGN WORKERS UNDER THE IMMIGRATION PROGRAM 


The departments’ response. Agreed. The departments have already 
taken steps to enhance program integrity and improve worker 
protections. For example, HRSDC has introduced new compliance 
review initiatives to better detect and address instances of 
non-compliance with conditions specified on labour market opinions. 
Information-sharing agreements with a number of provinces are either 
signed or forthcoming to support enforcement of federal and provincial 
laws and standards. In the spring, new policies were introduced that 
limit labour market opinion validity to six months and enable the 
revocation of labour market opinions when the conditions that 
supported that opinion are found to be no longer valid. Also, in 

June 2009, an electronic mailbox was created at HRSDC for CIC and 
Canada Border Services Agency staff to seek direction on specific 
labour market opinions issued by HRSDC, as well as providing a 
mechanism for reporting possible fraud. 


CIC and HRSDC continue to explore options for improving the 
Temporary Foreign Worker Program, including changes to the 
Immigration and Refugee Protection Regulations that would introduce 
enhanced monitoring of employers’ compliance with program 
requirements and permit refusal of service in instances where the 
requirements have not been met. 


2.118 In our April 2000 Report, we identified several problems with 
the efficiency and quality of application processing in missions 
overseas. Employees in offices abroad were buried in paperwork and 
spending a great deal of their time receiving and recording data for 
immigration applications, archiving files, handling correspondence, 
and collecting and reimbursing fees. 


2.119 We made several recommendations for improvement; the 
Department accepted them and outlined measures it would take 

to implement them. In our current audit, we therefore expected to 
find that the Department is now processing applications from foreign 
workers efficiently and has in place effective quality control and 
quality assurance procedures to ensure that decisions are consistent 
and comply with the Immigration and Refugee Protection Act and 
Regulations. 


Citizenship and Immigration Canada has improved some of its practices 


2.120 We found that Citizenship and Immigration Canada (CIC) has 
identified Client Service Modernization as one of its priorities 
since 2006. It developed a Client Service Improvement Framework to 
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guide the planning of initiatives and the development and delivery of 
solutions aimed at improving efficiency and client services. 


2.121 In addition, CIC has successfully implemented a number of 
initiatives and tools to address some inefficiencies in the processing of 
applications abroad. These include on-line validation of education 
diplomas in some missions, an increased use of third party language tests, 
and banking arrangements that remove the handling of cash from many 
missions. It has also implemented Visa Application Centres (VACs) in 
some countries to facilitate the visa application process for temporary 
resident applicants. VACs are private organizations that provide services 
on behalf of CIC. The aim is to increase client access to service, improve 
the quality of applications, and improve security at the missions by 
reducing in-person applications. We visited three missions where 
applicants can choose to use the services of a VAC. Staff at these 
missions said they appreciated the reduced paper burden and the quality 
of the applications they were receiving. CIC, however, has not yet 
conducted a formal cost-benefit assessment of these VACs. 


Delays in implementing new technologies have seriously limited efficiency gains 


2.122 In our 2000 Report, we noted that outdated technology in CIC 
offices abroad was a serious obstacle to improving efficiency. As part of 
its response to our recommendation, CIC stated, “Enhancements to 
our IT systems and infrastructure will significantly improve our ability 
to do business, both in terms of increased program integrity and better 
client service.” It also said that it was committed to developing and 
implementing its Global Case Management System (GCMS) to 
support its client case management operations. After a failed 

$80 million systems upgrade project initiated in 1994, CIC had just 
launched GCMS, which it viewed as the solution to support the 
Department’s entire client case management operations. 


2.123 Our 2006 audit of large information technology projects looked 
at GCMS. We found that CIC had to struggle with a number of 
problems, including changes in the project’s scope, difficulties in 
accessing funds, and a lack of people with the skills needed to manage 
the project. Various internal reviews had shown that GCMS was more 
complex than expected and faced unforeseen challenges. 


2.124 Although we did not re-examine the GCMS development 
process in this audit, we noted that in 2008 the government approved 
a reduced project scope focusing on Canadian immigration programs 
abroad. The estimated total cost of GCMS is now around $387 million, 
up from an original estimate of $194.8 million. Its implementation 
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in missions abroad is scheduled to begin in June 2010, almost 10 years 
after the initial launch of the project. We found that as a result of the 
delays in implementing this system, employees in offices abroad are 
still burdened by paperwork and spend a great deal of their time on 
clerical tasks. 


2125 Starting in 2011, the Department plans to implement in its 
overseas operations additional automated tools related to the 
processing of applications, including electronic applications and 
electronic payments. The Department told us that its goal is to create a 
client continuum—an electronic record that captures all immigration 
transactions for an individual—and shift to full electronic processing. 
Considering its past experience in system development, CIC will need 
to carefully manage all the risks associated with these projects to 
ensure that they will be completed on time and on budget. 


2.126 In our view, efficiency gains will be seriously limited until GCMS 
is implemented in missions abroad and CIC makes effective use of 
available technologies. 


A quality assurance framework has not yet been implemented 


2.127 In our 2000 Report, we also raised concerns about the quality of 
visa officers’ decisions on applications to immigrate to Canada. We 
stressed the need for a quality assurance framework to ensure that 
decisions are consistent and fair. 


2.128 The Department subsequently developed a quality assurance 
framework that is available to all missions, but there is no requirement 
for immigration program managers to implement it or to report on 
quality assurance. Consequently, while some quality assurance is 
performed in some missions when time permits, it is not comprehensive, 
not necessarily consistent with the framework, and does not ensure the 
overall quality and consistency of decisions. Of the eight missions we 
visited, only one had recently implemented the framework. 


2.129 Recommendation. Citizenship and Immigration Canada should 
ensure that its quality assurance framework is implemented fully and 


consistently in all missions. 


Department’s response. Agreed. Citizenship and Immigration Canada 
has recognized the importance of consistency in all of our program 
delivery. As we move to modernize our service delivery model we will 
ensure that quality assurance is a key objective of our program delivery 
and that it is applied in a consistent and risk-based fashion within 


two years. 
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Recognition of foreign credentials 


The federal government is contributing to the recognition of foreign credentials 


2.130 The importance of foreign credential recognition to the 
successful integration of immigrants into the labour market has been 
recognized for a long time.. The Immigration and Refugee Protection Act 
provides for the federal government to work in cooperation with the 
provinces and territories to achieve this objective. We therefore 
expected that Citizenship and Immigration Canada (CIC) and Human 
Resources and Skills Development Canada (HRSDC) would have 
implemented programs to meet the government’s commitments to 
facilitating foreign credential recognition. 


2.131 . Provincial and territorial governments have jurisdiction over 
the regulation of skilled trades and most professions. The federal 
government’s role is to provide strategic leadership to foster the 
development of a consistent nationwide approach to recognizing 
foreign credentials. 


2.132 In 2005, the federal government launched the Internationally 
Trained Workers Initiative, a comprehensive strategy for integrating 
Canadians and immigrants trained abroad into Canada’s labour force. 
The Foreign Credential Recognition program (FCRP) was part of this 
initiative; this contribution program delivered by HRSDC provides 
funding to provinces, territories, and stakeholders such as regulatory 
bodies, educational institutions, and the private sector to promote the 
recognition of foreign credentials for targeted occupations. 


2.133 The government also established a new organization within CIC, 
the Foreign Credentials Referral Office (FKCRO), which began operating 
in May 2007. Its purpose is to provide internationally trained and 
educated individuals with access to integrated, authoritative 
information about the Canadian labour market and credential 
assessment, recognition, and licensing processes and to provide 
path-finding and referral services. 


2.134 Exhibit 2.8 provides an overview of federal funding for these 
two initiatives over the last eight years. 


2.1385 HRSDC carried out an evaluation of the FCRP in 2008 to assess 
the program’s relevance, impacts, and cost-effectiveness. At the time 
of our audit, the evaluation report was being finalized. Preliminary 
findings indicate that the program is consistent with federal priorities 
and has contributed to a greater understanding and awareness of 
foreign credential recognition issues among stakeholders. 
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2.136 The FCRO informed us that in the 2008-09 fiscal year, foreign 
credential recognition information has been provided to over 
350,000 website visitors and that close to 38,000 in-person visits 

or telephone calls were made to Service Canada regional offices. 
The FCRO has put in place the necessary elements to carry out an 
evaluation of its performance, but it is too early in its operations to 
do so. 


A nationwide framework for recognition of foreign credentials is being developed 


2.137 In early 2009, first ministers and territorial leaders agreed to take 
specific action to develop a pan-Canadian framework for foreign 
qualification assessment and recognition by September 2009. In 

its 2009 Budget, the federal government committed $50 million over 
two years to support the development and the implementation of the 
framework. 


2.138 The Forum of Labour Market Ministers, led by HRSDC, is 
developing the framework. Its overall objective is to ensure that after 
submitting an application for licensing or registration, newcomers will 
receive a timely response as to whether their qualifications will be 
recognized or what additional requirements they must meet for 
registration, or they will be directed to alternative pathways 
commensurate with their skills and experience. 


Exhibit 2.8 The federal government contributes funding to programs that foster recognition of foreign credentials ($ millions) 


Foreign Credential 
Recognition Program 


- 2003-04 2004-05 2005-06 2006-07 2007-08 


2008-09 2009-10 


17.4 96-9 


Foreign Credentials 
Referral Office* 


13.9 27.6 


* Includes funding provided to Human Resources and Skills Development Canada (HRSDC) for service delivery in Canada. The Office began operating in May 2007. 


Source: HRSDC and Citizenship and Immigration Canada 
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Conclusion 


2.139 Overall, we found that current practices of Citizenship and 
Immigration Canada (CIC) and Human Resources and Skills 
Development Canada (HRSDC) do not ensure that foreign worker 
programs are delivered efficiently and effectively. 


2.140 Although the process followed by CIC to design the Canadian 
Experience Class category was very good, other key decisions were 
made in recent years without a thorough analysis of their costs and 
benefits, risks, and potential impacts on other programs and delivery 
mechanisms. Program changes have caused a significant shift in the 
types of workers being admitted permanently to Canada under the 
Economic class. Until it develops a strategic roadmap for the future 
and it evaluates the performance of its current programs, CIC will not 
be in a position to demonstrate that its programming best meets the 
needs of the Canadian labour market. 


2.141 Managing the large inventory of applications under the Federal 
Skilled Worker (FSW) category is a major challenge. Measures taken 
by CIC in 2008 to limit the number of new Federal Skilled Worker 
applications were implemented without sufficient analysis. While it is 
too early to assess their full impact, the trends in the number of new 
applications received since the beginning of 2009 indicate they might 
not have the desired effect. CIC will have to monitor the situation 
closely and might need to consider other strategies to achieve this 
objective. Failure to do so could result in CIC being unable to process 
new applications within the 6 to 12 months it has forecast. In addition, 
the Department's ability to reduce the inventory of some 

635,000 applicants on hand at the end of February 2008 could be 


significantly impaired. 


2.142 The issues we noted in the delivery of the Temporary Foreign 
Worker Program pose significant risks to the integrity of the program 
and could leave many foreign workers in a vulnerable position, 
particularly those who are physically or linguistically isolated from the 
general community or are unaware of their rights. In particular, the fact 
that HRSDC and CIC have not defined their respective roles and 
responsibilities in assessing the genuineness of job offers could result in 
work permits being issued for employers or jobs that do not exist. 
There is also no systematic follow-up by either CIC or HRSDC to 
ensure that in their previous and current employment of temporary 
foreign workers, employers have complied with the terms and 
conditions (such as wages and accommodations) under which the 
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work permits were issued. Finally, HRSDC’s practices in issuing labour 
market opinions do not ensure the quality and consistency of decisions. 


2.143 In its processing of applications in missions overseas, CIC has 
successfully introduced a number of initiatives and tools to address 
some of the inefficiencies we noted in 2000. However, an information 
technology system that is key to its plans has been under development 
for almost 10 years. As a result, employees in offices abroad are still 
buried in paperwork and spending a great deal of their time on clerical 
tasks. In addition, the Department has not yet implemented a quality 
assurance framework to obtain assurance that decisions made by its 
visa officers are fair and consistent. 
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About the Audit 


All of the audit work in this chapter was conducted in accordance with the standards for assurance 
engagements set by The Canadian Institute of Chartered Accountants. While the Office adopts these 
standards as the minimum requirement for our audits, we also draw upon the standards and practices of 
other disciplines. 


Objective 


The audit objective was to determine whether Citizenship and Immigration Canada (CIC) and Human 
Resources and Skills Development Canada (HRSDC) efficiently and effectively handle program planning 
and delivery to facilitate the entry of permanent and temporary foreign workers into Canada. 


Scope and approach 


Our audit covered the planning and management of the following programs since the introduction of 


the Immigration and Refugee Protection Act (IRPA) in 2002: 
¢ Federal Skilled Worker, 
« Canadian Experience Class, 
- Live-in Caregiver, 
¢ Provincial Nominee Program, 
- Temporary Foreign Worker Program, 
¢ Foreign Credentials Referral Office, and 


- Foreign Credential Recognition Program. 


In addition, we followed up on some observations and recommendations that we made in our April 2000 
report, Chapter 3, Citizenship and Immigration Canada—The Economic Component of the Canadian 
Immigration Program. 


At CIC, we interviewed key people involved in processing applications from foreign workers, reviewed 
files, records, and reports, and analyzed management information databases. We also conducted audit 
work at the following missions abroad: Beijing, Buffalo, London, Manila, Mexico, New Delhi, Paris, and 
Port-of-Spain. 


At HRSDC, we interviewed officers and managers in four regions—Vancouver, Edmonton, Toronto, and 
Montreal—and at HRSDC’s headquarters, and we reviewed files, records, and reports. Furthermore, we 
examined a statistical sample of 99 labour market opinion (LMO) applications across Canada for the 
period from 1 October 2007 to 30 September 2008. 
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Criteria 


Listed below are the criteria that were used to conduct this audit and their sources. 


Criteria ; 


eee Planning of programs 


We expected that Citizenship and Immigration Canada (CIC) 


would have adequately planned their programs to facilitate 
the entry of foreign workers into Canada. 


and Human Resources and Skills Development Canada (HRSDC) 


Sources 


* Immigration and Refugee Protection Act, sections 3.(1)(a) 
(c), and (f), and 87.3(2) 


¢ Treasury Board of Canada Secretariat, Management 
Accountability Framework, Round V—Stewardship (2007) 


Inventory 


We expected that CIC would have managed the inventory of 
applications for federal skilled workers in a way that ensures 
processing within a reasonable time period as defined by CIC. 


¢ Immigration and Refugee Protection Act, section 87.3 


* Treasury Board of Canada Secretariat, Management 
Accountability Framework, Round V—Stewardship (2007) 


Processing of applications 


We expected that CIC would have efficiently processed 
applications from foreign workers, and that HRSDC would have 
efficiently processed applications for labour market opinions. 


e Treasury Board of Canada Secretariat, Management 
in the Government of Canada: A Commitment to Continuous 
Improvement (2004) 


Treasury Board of Canada Secretariat, Management 
Accountability Framework, Round V—Stewardship (2007) 


Citizenship and Immigration Canada’s, operations manual, 
OP 1—Procedures, 5.14, Processing priorities 


Treasury Board of Canada Secretariat, Service Standards: 
A Guide to the Initiative 


Treasury Board, Policy on Service Standards for External 
Fees (2004) 


We expected that CIC and HRSDC would have effective quality 
control and assurance procedures in place to ensure that 
decisions are consistent and in compliance with the /mmigration 
and Refugee Protection Act and Regulations, and operational 
procedures in processing applications. 


] 
¢ /mmigration and Refugee Protection Act, 3(1)(f) and 3(3)(d) 


¢ April 2000 Report of the Auditor General of Canada, 
Chapter 3, Citizenship and Immigration Canada— 
The Economic Component of the Canadian Immigration 
Program 


Citizenship and Immigration Canada, operations manual 
FW 1 Procedures for Temporary Foreign Workers, Appendix J, 
page 171 


* Citizenship and Immigration Canada, operations manual 
OP 1—Procedures, 9.2 


Foreign credential recognition 


We expected that CIC and HRSDC would have implemented 
programs to meet the government's commitments to facilitating 
foreign credenitial recognition. 


¢ /mmigration and Refugee Protection Act, section 3(1)(j) 

¢ Federal Accountability Act, 2006 

* Budget Plan 2006 

* Department of Finance Canada, Advantage Canada, 2006 


¢ Treasury Board of Canada Secretariat, Management 
Accountability Framework, Round VI—Policy and 
programs (2008) 


Management reviewed and accepted the suitability of the criteria used in the audit. 
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Period covered by the audit 


With the exception of the Foreign Credential Recognition line of enquiry, whose period audited started at 
the launching of the Internationally Trained Workers Initiative in 2005, and the review of labour market 
opinion applications, which was based on a statistical sample of applications processed between 

1 October 2007 and 30 September 2008, the period audited for this Chapter goes from the coming into 
effect of the Immigration and Refugee Protection Act in June 2002 until 30 June 2009. 


Audit work for this chapter was substantially completed on 30 June 2009. 


Audit team 
Assistant Auditor General: Richard Flageole 


Principal: Suzanne Therrien 
Lead Director: Jean Goulet 


Lucie Després 
Marc Gauthier 
Yan Lehoux 
Philippe Malette 
Denis Roy 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 2. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


Strategic planning and programming 


2.34 ‘Io strengthen its strategic 
planning, Citizenship and Immigration 


Canada (CIC) should 


¢ develop, in consultation with 
provinces and territories, a strategic 
roadmap for the future of the 
immigration program including a 
clear vision of what each category 
is expected to contribute, on a 
multi-year basis, toward achieving 
the objectives of the Economic class; 
and 


* take the necessary steps to ensure 
that its programming is optimal and 
closely aligned with that vision. 


(2.24-2.33) 


2.39 Citizenship and Immigration 
Canada and Human Resources and 
Skills Development Canada (HRSDC) 
should carry out evaluations of 
permanent and temporary foreign 
worker programs according to their 
approved evaluation plans. 
(2.35-2.38) 


Agreed. Work is already underway that will contribute to 
the vision and future of the immigration program. A notable 


example is the Department’s work on the broader modernization 
of the immigration program, including measures that have 
policy, operational, and service improvement and technological 
components. In addition, the evaluation of the Federal Skilled 
Worker program and work on regulatory changes to the 
Temporary Foreign Worker Program to address concerns around 
program integrity are two more current areas of activity that 
support the modernization of the immigration system. Taken 
together, this work will lead to the development of an 
immigration roadmap, which will be developed in consultation 
with our provincial and territorial partners over the next two 
years. 


In terms of the balance between categories in the Economic 
class, Citizenship and Immigration Canada launched a process in 
November 2008 with provinces and territories to develop a 
common approach for understanding how various programs 
under the Economic class, such as Federal Skilled Worker and 
Provincial Nominee, contribute to meeting the economic 
objectives for immigration, such as responding to labour-market 
needs. Part of this effort involves achieving an appropriate 
balance among programs. 


Agreed. The departments have initiated work on a joint, 
comprehensive evaluation of the Temporary Foreign Worker 
Program that will provide results in 2010-11. CIC is currently 
carrying out a comprehensive evaluation of the Federal Skilled 
Worker program; other programs in the permanent stream will 
be evaluated according to the CIC evaluation plan. 
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Federal skilled workers 


2.56 In developing future ministerial 
instructions, Citizenship and 
Immigration Canada should ensure that 
selected strategies are supported by 
strong policy, program, and operational 
analysis. (2.53-2.55) 


2.62 Citizenship and Immigration 
Canada should closely monitor the 
impact of the new eligibility criteria on 
the number of applications to be 
processed under the Federal Skilled 
Worker category and on the 
effectiveness of its inventory reduction 


strategy. (2.57-2.61) 


2.68 Citizenship and Immigration 
Canada, in consultation with key 
partners, should put in place structures 
and processes to ensure that the 
selected eligible occupations remain 
relevant to the changing needs of the 
Canadian labour market. (2.63—2.67) 


Agreed. The Department agrees with the importance placed on 
strong policy analysis in the issuance of future ministerial 
instructions. . 


To this end, the Department is working on several initiatives, 
including ensuring that diagnostic and analytical tools are in 
place for the monitoring and evaluation of the current 
ministerial instructions, improved analytical capacity around 
labour market information, and the exploration of options for 
expert advice to be regularly available to Citizenship and 
Immigration Canada regarding Canadian labour markets and 
current and projected occupational shortages. 


Agreed. The Department has introduced measures to ensure 
timely and comprehensive monitoring of the impact of the 
ministerial instructions. First, using existing data sources, the 
Department regularly tracks and analyzes data relating to, 
among other things, application intake rates, acceptance rates, 


‘and outcomes at key decision points throughout the processing 


continuum (i.e., from the Centralized Intake Office to processing 
and visa issuance at overseas missions). Second, the Department 
has introduced (effective June 2009) monthly reporting 
templates for overseas missions to capture more qualitative 
information on how the ministerial instructions are affecting the 
Federal Skilled Worker category. Taken together these initiatives 
will inform the need to revise ministerial instructions as 
appropriate. 


Agreed. Timely and accurate labour market information is an 
important component of ensuring that the ministerial 
instructions help meet the Government of Canada’s goal of 
improving the labour market responsiveness of the immigration 
program. Over the next two years, Citizenship and Immigration 
Canada will work with key partners to explore options for 
structures and processes to better ensure the instructions meet 
their objectives. Part of this work includes exploring options for 
an external advisory body, such as the U.K.’s Migration Advisory 
Committee. | 
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2.78 Citizenship and Immigration Agreed. The Department intends to continue to move on its 
Canada should evaluate, after the first modernization agenda as quickly as possible with the support of 
two years of operations, the cost- the roll-out of our Global Case Management System. Within our 
effectiveness of the Centralized Intake vision of moving some work back to Canada and working in a 
Office and the extent to which it is more virtual network, the Centralized Intake Office operations 
meeting its objectives. (2.69-2.77) will be evaluated and a decision made on its cost-effectiveness 


and its support to meet CIC overarching objectives. The 
evaluation of the Centralized Intake Office is included in the 
Department’s Evaluation Plan in 2010/2011. 


Provincial Nominee Program 


2.85 Citizenship and Immigration Agreed. The Department has developed an evaluation 
Canada should work with provinces framework and logic model for an evaluation of the Provincial 
and territories to ensure that, for the Nominee Program, which is in the Department’s Evaluation 
Provincial Nominee Program, Plan for 2010/11. Common performance measures and 


indicators will be drawn from the longitudinal immigration 
¢ mechanisms are in place to collect 


and share appropriate information 
against agreed upon evaluation 


database, which will provide information related to outcomes 
and retention for all provinces and territories. The Department 
remains open and committed to working with provinces and 


criteria; a ae : 
territories on any additional information sources and 


evaluation plans are prepared and mechanisms they wish to have considered. 
implemented to assess whether 


eee The evaluation will assess performance of provincial nominee 
provincial nominee programs are 


programs against the objectives of the Immigration and Refugee 


meeting the objectives set out in the 
° Protection Act and Regulations. 


Immigration and Refugee Protection Act 
and Regulations; and Citizenship and Immigration Canada agrees with the need for 
quality assurance mechanisms and is open to working with 
provinces and territories to develop them. Should there be 
questions related to quality assurance, CIC can substitute its 
own evaluation of provincial and territorial nominations. 


* quality assurance mechanisms are in 
place to ensure that nomination 
decisions are consistent and 
compliant with the Immigration and 
Refugee Protection Act and Regulations 
and their respective Provincial 
Nominee Program criteria. 


(2.80-2.84) 
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The Temporary Foreign Worker Program 


2.101 Human Resources and Skills Agreed. The Department has already taken steps to build on the 
Development Canada should program efficiencies achieved with Budget 2007 funding that 
will further enhance the effectiveness of processing labour 
market opinions. In this regard, the Department has put in place 
mechanisms to strengthen coordination with regional staff to 
enhance integrity, operations, and service delivery. This includes 


¢ provide clear directives, tools, and 
training to officers engaged in issuing 
labour market opinions; and 


¢ implement a quality assurance working with regions to develop new directives and clearer 
framework to ensure the quality and _ bulletins to improve national consistency in operations and 
consistency of opinions across decision-making. In March 2009, an electronic mailbox was 
Canada. (2.96—2.100) created for regional staff to seek direction and exchange 


information on issues related to policy and operations. The 
Department continues to hold annual coordination meetings 
and monthly teleconferences between national headquarters and 
Service Canada regions to discuss operational and procedural 
issues. In July 2009, a new monthly teleconference was launched 
specific to integrity issues. 


In June 2009, a new team was created to develop and implement 
a quality assurance framework that will strengthen the 
responsiveness, consistency, speed, and quality of decision- 
making when issuing labour market opinions. This initiative will 
also help to improve overall program integrity. The team is 
working closely with headquarters and regional staff to ensure 
the framework is informed by an understanding of the 
operational realities faced in each region, and can therefore be 
an effective tool to support national consistency in decision- 
making. 
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SELECTING FOREIGN WORKERS UNDER THE IMMIGRATION PROGRAM 


2.107 Citizenship and Immigration © 
Canada and Human Resources and 
Skills Development Canada should 
clarify their respective roles and 
responsibilities, and put mechanisms in 
place to ensure that the genuineness of 
job offers is systematically verified. 


(2.102-2.106) 


2.117 Citizenship and Immigration 
Canada and Human Resources and 
Skills Development Canada should 
implement mechanisms that would 
better enable thém to ensure the 
integrity of the Temporary Foreign 
Worker Program and the protection of 
individuals. (2.108—2.116) 


Agreed. The departments are working together to make 
improvements in these areas. 


The departments are also developing regulatory options to 
strengthen the integrity of the Temporary Foreign Worker 
Program including clarity around roles and responsibilities. Part 
of this work focuses on more structured assessments of the 
genuineness of both employers and jobs offered to temporary 
foreign workers. A new information-sharing agreement between 
CIC and HRSDC will enable a more active exchange of 
information pertinent to the assessment of the genuineness of 
job offers. 


Additionally, new interdepartmental working groups and 
information-exchange mechanisms have been established to 
ensure better communication and coordination of program 
development and implementation. 


Agreed. The departments have already taken steps to enhance 
program integrity and improve worker protections. For example, 
HRSDC has introduced new compliance review initiatives to 


better detect and address instances of non-compliance with 


conditions specified on labour market opinions. Information- 
sharing agreements with a number of provinces are either signed 
or forthcoming to support enforcement of federal and provincial 
laws and standards. In the spring, new policies were introduced 
that limit labour market opinion validity to six months and 
enable the revocation of labour market opinions when the 
conditions that supported that opinion are found to be no longer 
valid. Also, in June 2009, an electronic mailbox was created at 
HRSDC for CIC and Canada Border Services Agency staff to 
seek direction on specific labour market opinions issued by 
HRSDC, as well as providing a mechanism for reporting possible 


fraud. 


CIC and HRSDC continue to explore options for improving the 
Temporary Foreign Worker Program, including changes to the 
Immigration and Refugee Protection Regulations that would 
introduce enhanced monitoring of employers’ compliance with 
program requirements and permit refusal of service in instances 
where the requirements have not been met. 
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Processing of applications 


2.129 Citizenship and Immigration 
Canada should ensure that its quality 
assurance framework is implemented 
fully and consistently in all missions. 


(2.127-2.128) 


Chapter 2 


Agreed. Citizenship and Immigration Canada has recognized the 
importance of consistency in all of our program delivery. As we 
move to modernize our service delivery model we will ensure 
that quality assurance is a key objective of our program delivery 
and that it is applied in a consistent and risk-based fashion. 
within two years. 
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What we examined 


Why it’s important 


Income Tax Legislation 


Main Points 


The Department of Finance Canada is responsible for formulating tax 
policy at the federal level and for developing the legislation and 
regulations needed to implement federal tax measures. In the past, the 
Department has developed legislative amendments to address 
technical deficiencies in the Income Tax Act, in order to more clearly 
convey the intent of the legislation or to make it easier for taxpayers to 
comply with it. Packages of technical amendments have first been 
released for public comment and then tabled in Parliament in the form 
of a technical bill. Usually these amendments do not involve any 
reconsideration of tax policy. 


The Canada Revenue Agency is responsible for applying and 
interpreting the Income ‘Tax Act—one of the longest and most complex 
pieces of Canadian legislation. The primary goal of the Agency as 
Canada’s tax administrator is to ensure that taxpayers comply with 
their obligations and that Canada’s tax base is protected. 


We looked at how the Department of Finance Canada develops 
technical amendments for tabling in Parliament; we did not examine 
the process for developing legislative amendments that involve 
changes in tax policy. 


In addition, we examined how the Canada Revenue Agency provides 
taxpayers and its tax auditors with guidance on the application and 
interpretation of the Income Tax Act, and how it assists the Department 
of Finance Canada in identifying and developing technical changes 
that may be needed in the legislation. The audit covered the period 
from 2004 to 2009, with relevant historical information reviewed back 


to 1973. 


Canada’s tax system relies on taxpayers to self-assess and pay the 
income taxes they owe. According to the Canada Revenue Agency, 
most taxpayers will meet their tax obligations if given the proper tools 


and information. 


Taxpayers’ ability to comply with tax legislation depends on. their 
understanding of how the rules apply to their own circumstances. 
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What we found 


Chapter 3 


When the intent of the legislation is not clearly conveyed by the 
words, taxpayers may find it difficult to assess the income taxes they 
owe and this could foster tax avoidance. Uncertainty about how the 
law should be applied can also add to the time taken and costs incurred 
by tax audits and tax administration. 


« No income tax eau anal has been passed since 2001. Although 


the government has said that an annual technical bill of routine 
housekeeping amendments to the Act is desirable, this has not 
happened. As a result, the Department of Finance Canada has a 
backlog of at least 400 technical amendments that have not been 
enacted, including 250 “comfort letters” dating back to 1998, 
recommending changes that have not been legislated. 


Department of Finance Canada staff are well-informed about the 
identified issues, but they do not make effective use of available 
electronic tools to record, track, and prioritize them for possible 
legislative amendment. 


The Agency’s processes for tracking and monitoring issues in the Act 
are not adequate. The Agency has no formal database to keep track 
of the identified legislative issues and their disposition. Issues are 
validated, and analysis of some of them is carried out. 

However, there is no systematic review of their impact on 
compliance or on the tax base. As part of the annual Budget process, 
the Agency selects a relatively small number of outstanding issues as 
priorities for communication to the Department, based on the 
assessment of senior Agency officials. 


- The Agency provides guidance to taxpayers and tax practitioners in 


the form of advance income tax rulings, technical interpretations, 
Income Tax Interpretation Bulletins, as well as tax guides and 
pamphlets, to name a few. Tax practitioners we interviewed value the 
rulings process. However, in the past three years, the Agency has not 
met its timeliness standard for issuing advance income tax rulings. In 
addition, some of the information in the Agency’s Income Tax 
Interpretation Bulletins is no longer current and the public is not 
always made aware of that fact. 


The Agency and the Department have responded. The Agency and 
the Department agree with all of our recommendations. Their detailed 
responses follow each recommendation throughout the chapter. 
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Introduction 


The need for clarity in tax legislation 


3.1. The Income Tax Act is the central component of the federal 
income tax system, and it sets out the rules for compliance and 
provides for the administration of the Act. Other elements of this 
system include the Income Tax Regulations, tax treaties, and court 
decisions. For the system to work for both tax administrators and 
taxpayers, the Act’s clarity must be maintained. 


3.2. Our system of income taxation depends on taxpayers 
self-assessing their tax obligations based on a clear understanding of 
the law. For many tax filers, the main guidance for self-assessment is 
the income tax form and tax guides that are prepared by the Canada 
Revenue Agency. 


3.3. When the intent of the statute is not clearly conveyed by its 
words, Canadians may not be able to easily self-assess or correctly 
calculate their taxes. Where necessary, tax administrators will offer 
additional guidance to Canadians on how tax law is interpreted and on 
how it is applied to their circumstances. Legislative drafters in the 
Department of Finance Canada may propose amendments to sections 
of the Act, for Parliament to approve, if lack of clarity compromises 
Canadians’ ability to comply or the Act does not adequately reflect the 
underlying policy. These proposed amendments are first released to the 
public for comment and then tabled in Parliament in a technical bill or, 
occasionally, in a budget bill. These amendments may be relieving 
(resulting in less tax) or tightening (resulting in more tax). 


3.4 The Supreme Court of Canada has said that tax laws should be 
certain, predictable, and fair so that taxpayers can order their affairs 
intelligently. 


3.5 The application of tax provisions may become uncertain for a 
number of reasons. For example, a court decision may overturn an 
interpretation established by the Canada Revenue Agency. The 
Department may indicate the intent to propose legislative changes, but 
taxpayers will not know the exact nature of the proposed changes until 
they are enacted. Taxpayers may be uncertain when they file the 
current year’s tax return, and they may have to choose between filing 
under the current legislation or the proposed legislation. New forms of 
business transactions may also challenge the application of the current 
provisions. Another type of uncertainty arises if a proposed 
amendment is not enacted for more than a year, but is to be passed 
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with retrospective effect (that is, only from the current date but 
affecting arrangements from before that date). Retrospective 
amendments can change the tax result and require a change in the tax 
assessment issued by the Canada Revenue Agency. i 


3.6 For taxpayers, the negative effects of uncertainty may include 


¢ higher costs of obtaining professional advice to comply with tax 
law; 


¢ less efficiency in doing business transactions; 


¢ inability of publicly traded corporations to use proposed tax | 
changes in their financial reporting, because they have not been 
« D ” 
substantively enacted”; 


* greater cynicism about the fairness of the tax system; and 


* increased willingness to use aggressive tax plans. 


3.7 For the tax administrator, the negative effects may include 


¢ higher costs for providing additional guidance and interpretations 
to taxpayers and tax auditors; and 


¢ higher administrative costs for reprocessing the tax returns after 
an outstanding legislative amendment is enacted and for 
obtaining waivers to extend the limitation period for 
reassessment. 


The result may be uncertainty in the amount of tax revenues to be 
collected by the government and possible loss of tax revenues. 


3.8 If proposed technical changes are not tabled regularly, the 
volume of amendments becomes difficult for taxpayers, tax 
practitioners, and parliamentarians to absorb when they are grouped 
into a large package. 


The roles of the Agency and the Department in providing clarity in tax legislation 


3.9 Canada Revenue Agency. The Agency is responsible for 
administering the Income Tax Act and its regulations and related 
statutes, for collecting taxes, and for interpreting tax law. The Agency 
provides information to the Department of Finance Canada about 
compliance challenges and aims to make the Department aware of the 
need for technical amendments to the Act that it has identified. In 
order to give the correct legislative guidance, the Agency consults with 
the Department if the underlying policy intent of a provision needs to 
be better understood. 
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Comfort letters—Letters in which the 
Department of Finance Canada promises to 
recommend to the Minister that a minor, 
non-policy legislative change be made; usually 
these changes are made to provide relief to the 
taxpayer when the tax result is clearly not the 
one intended by the law. 
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3.10 The Agency provides advice and guidance on the interpretation 
and application of the Income Tax Act to taxpayers and its tax auditors, 
as follows: 


¢ It gives general information in guides and pamphlets, instructions 
on tax forms, and guidance through its telephone service and 
website. 


¢ It gives more specific guidance through a series of Income Tax - 
Interpretation Bulletins and Information Circulars, and through a 
publication called Income Tax Technical News. 


¢ It gives advance income tax rulings on the application of the law 
to specific transactions proposed by taxpayers, upon request, for a 
fee. The Agency regards these advance income tax rulings as 
binding. It also gives non-binding tax interpretations and opinions 
to taxpayers who request them and to tax auditors in the midst of 
an audit, if the auditor and the taxpayer do not agree on the 
interpretation of a provision in the Act. 


3.11 Department of Finance Canada. The Department is responsible 
for developing and evaluating federal tax policy. It is responsible for 
drafting legislation that clearly reflects the tax policies set by the 
federal government. It also advises the Minister of Finance on the need 
for technical changes and policy changes to income tax legislation to 
ensure that Canada’s tax system functions effectively. The Department 
of Finance Canada is also responsible for developing technical 
amendments to income tax legislation. Technical amendments are 
changes made to correct anomalies that arise after the original measure 
was passed and to correct consequences that were not intended. These 
amendments are not intended to introduce new tax policy or change 
existing tax policy. 


3.12 Although it is not required to do so, the Department also issues, 
upon request and where appropriate, comfort letters to taxpayers. 


Focus of the audit 


3.13. The audit focused on activities within the Canada Revenue 
Agency and the Department of Finance Canada that identify and 
develop legislative amendments to correct technical deficiencies. It 
also focused on Agency activities that provide guidance to taxpayers 
and tax auditors on how to interpret and apply tax law. For purposes of 
this audit, we did not examine the process that seeks to bring about 
new policy changes through amendments to the Act or the more 
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The Department’s legislative 
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maintenance 


general guidance provided by the Agency, such as tax guides, 
pamphlets, phone service, and its website. 


3.14. We consulted 54 tax practitioners from across Canada who were 
referred to us by the Canadian Tax Foundation (an independent tax 
research organization) to find out 


¢ how they use advance income tax rulings and income tax 
technical interpretations, issued by the Agency, and comfort 
letters, issued by the Department; 


¢ how useful and timely they find the Agency’s current legislative 
guidance; and 


¢ what they see as the impact of technical legislative issues that 
have not been addressed. 


3.15 We interviewed tax auditors who use guidance on legislative 
interpretation and application from the Agency and headquarters staff 
who provide it to them. We reviewed management information, where 
available, on these activities that describe the level of service. 


3.16 We interviewed Agency officials who are responsible for 
identifying legislative issues and Department officials who are 
responsible for developing the technical amendments to address them. 
We reviewed their processes, inventories, and production levels. We 
also reviewed the process by which Department of Finance Canada © 
provides comfort letters to taxpayers and then produces technical 
amendments. We have included five case studies to illustrate our 
observations. These appear throughout the chapter. 


3.17. More details on the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


The Department’s methods to record and track legislative issues are not integrated 


3.18 The Tax Legislation Division of the Department of Finance 
Canada receives information about technical issues in income tax 
legislation from the Canada Revenue Agency. It also receives 
comments and questions on tax laws from taxpayers; tax practitioners; 
tax groups, such as the Joint Committee of the Canadian Bar 
Association and The Canadian Institute of Chartered Accountants; 
and the Tax Executives Institute as well as from industry associations. 
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On average, the Division receives 125 written submissions a year, 
relating to many sections of the Income Tax Act. 


3.19 One of the functions of the Division (and of other divisions 
within the Tax Policy Branch, the Agency, and the Department of 
Justice Canada) is to analyze instances of the federal government 
losing a tax case. The Division’s role is to develop technical changes 
necessary to more clearly convey the policy intent of the income tax 
legislation, as it relates to the topic that was before the courts. 


3.20 We examined whether the Department had a well-managed 
process in place to comprehensively record, track, and prioritize 
legislative issues, and whether it used this process to select issues for 
draft technical amendments. We found that Tax Legislation Division 
staff is well-informed about the issues, but the Department relies on 
basic, people-dependant processes. It does not use its available 
electronic tools effectively. 


3.21 The identified technical issues sent to the Department are 
recorded in several ways. 


3.22. Correspondence log. The Tax Legislation Division maintains a 
log of incoming letters and tracks progress in their responses. Some of 
these letters point out legislative issues that will need to be dealt with 
by amendments. Our analysis of the log shows that 74 percent of the 
letters received between 2004 and 2008 remain open and incomplete. 
Department officials told us that most of the letters did not identify 
technical issues that required legislative changes. 


3.23 Electronic database. In the early 1990s, the Department 
developed an electronic listing of possible legislative issues to replace 
paper versions. This electronic tool was intended to help set priorities, 
allocate resources, and help officers’ understanding both of the issues 
involved and the reasons for amending the tax legislation. This 
electronic tool can provide a historical record and searchable database 
of all issues that the Department has identified. It can also help the 
Department rank the numerous issues by priority. 


3.24 Despite the important features of this electronic database and 
the fact that it was introduced more than 15 years ago, it is not being 
used consistently in the Tax Legislation Division. Since its inception, 
hundreds of possible technical issues have been entered into the 
database; however, only two items have been recorded in the last year. 
Division officers and chiefs maintain and rely primarily on their own 
individual parallel lists of legislative issues. Rather than using the 
electronic database to maintain an up-to-date and prioritized 
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inventory, the practice in the division is to manually consolidate the 
chiefs’ lists with the officers’ lists periodically. A partial consolidation 
in October 2008 identified 178 outstanding technical deficiencies from 
these lists. 


3.25 Comfort letters. When the Department sends a comfort letter 
in response to a taxpayer request, it keeps a paper copy in a binder. 
Over the years, some of the issues identified in comfort letters have 
been entered into the electronic inventory. The inventory contains an 
incomplete electronic summary of these letters and their legislative 
issues. 


3.26 Department officials told us that they use their professional 
judgment to determine which legislative deficiencies should be a 
priority for the development of draft legislation. No formal criteria 
exist for identifying priority legislative amendments, but Department 
officials attach broad priority rankings (low, medium, high) to 
particular items as they arise. They consider issues raised in comfort 
letters to be high priority. — 


3.27 Recommendation. The Department of Finance Canada should 
use an integrated and consistent process for recording, tracking, and 
prioritizing all technical issues for possible legislative amendment. 


The Department’s response. Agreed. While systems are in place to 
manage and keep track of technical issues for possible legislative 
amendment, improvements can be made with regard to database 
management and usage. The Department will undertake, as resources 
permit, to consolidate its system for ensuring that technical issues are 
documented and catalogued consistently and that this system is 
maintained and kept up to date as new issues arise, and as existing 
issues are addressed. 


The last income tax technical bill was passed by Parliament in 2001 


3.28 The concept of a tax bill that would be separate from the federal 
budget and would provide technical amendments to the Income Tax 
Act first arose in 1978, when the Canadian Tax Foundation set up a 
committee on the federal budget process. In 1982, the government 
adopted a committee recommendation to propose non-policy 
amendments in an annual technical bill. The government produced 
these technical bills for a number of years. 
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3.29 In the 1991 Report of the Auditor General, Chapter 2, we 
expressed concerns that income tax comfort letters were given without 
public announcement. In response, the Department of Finance 
Canada stated that “the government intends to release a package of 
income tax technical amendments on an annual basis, so that 
taxpayers will not be subject to more lengthy waiting periods as in the 
past before amendments are released to the public.” While comfort 
letters have since been regularly released to the public, very few 
technical bills have been introduced or passed in recent years. In the 
last 18 years, only four such income tax bills have been enacted 


(Exhibit 3.1). 


Exhibit 3.1 Only four income tax technical bills have been enacted since 1991 


Year the bill was introduced Bill number Date of Royal Assent 
1992 Bill C-92/C-15 12 May 1994 
1993 Bill C-27. 18 June 1994 
1995 to 1997 | Bill C-28 (Div B) 18 June 1998 
2001 Bill C-22 (Part 1) | 14 June 2001 


Source: OAG audit files 


3.30 The most recent package of technical amendments was released 
to the public on 20 December 2002. Between 2002 and 2006, more 
measures were added to the package. Some amendments, including 
those regarding foreign investment entities and non-resident trusts, 
were considered to be changes in policy. The technical portion of this 
bill consisted of 282 pages, with 155 clauses of amendments, to address 
legislative deficiencies dating back to 1996. The package was tabled as 
a bill in the House of Commons in November 2006, but it died when 
Parliament was prorogued in 2007, and an identical bill tabled in 2007 
died when Parliament was dissolved in 2008 (Exhibit 3.2). As of the 
date of this audit, the technical amendments from this bill have not 
been reintroduced in the House of Commons. Two examples of the 
effects of a delay in technical amendments are in the two case studies 


that follow Exhibit 3.2. 
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Exhibit 3.2 The one technical bill that was tabled in Parliament since 2001 has not been enacted 


- Chronology of the Bill 


20 December 2002 


Minister of Finance Canada released for consultation a 
package of technical amendments. 


27 February 2004 


Minister of Finance Canada released a revised package of 
technical amendments that included the December 2002 
amendments and additional ones developed since 2002. 


18 July 2005 


Minister of Finance Canada released for consultation a revised 
package of technical amendments. 


29 November 2006 


A Notice of Ways and Means Motion to introduce a bill with the 
technical amendments was tabied in the House of Commons. 
The Bill also included rules regarding foreign investment 
entities and non-resident trusts. This became Bill C-33. 


15 June 2007 


Bill C-33 received third reading in the House of Commons. 


14 September 2007 


Parliament was prorogued; Bill C-33 died on the order paper. 


29 October 2007 


Bill C-10 was introduced in the House of Commons. It was 
identical to Bill C-33. 


December 2007 to 
June 2008 


The Senate Standing Committee on Banking, Trade and 
Commerce held several hearings to discuss Bill C-10. 


7 September 2008 


Parliament was. dissolved; Bill C-10 died on the order paper. 


‘As of the date of substantial completion of our audit, 24 June 2009, technical 
amendments from Bil! C-10 had not been reintroduced. 


Sources: 1. For bills tabled prior to 29 January 2001, see Parliament of Canada House Bills website. 
2. For bills tabled on or after 29 January 2001, see Parliament of Canada LEGISinfo website. 


: Routine technical change 


This case study illustrates the key problem identified in this chapter: 14 years after the 
need for a non-controversial technical amendment was identified, it still has not been 


made. 


Many professionals and small businesses operate as a partnership. Under existing rules 
in the Income Tax Act, when a partner leaves a partnership at any time other than at the 
start of the partnership’s fiscal year, the timing for calculating income and recognizing 
capital gains does not match. This can lead to the exiting partner paying double tax. 


Before 1995, the Agency was able to administer tax law so that no double tax would be 
paid. However, in 1995 the Agency had to change its way of interpreting the law 
because new rules came into force in the Act. 


The Department of Finance Canada released draft legislation to correct this problem in 
2002. The Canada Revenue Agency has been applying the draft legislation so that 
double taxation is avoided. However, the draft legislation originally tabled in Bill C-33 
has not yet been passed (see Exhibit 3.2). 


Source: OAG audit files 
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Statute-barred—tThe date beyond which the 
Minister may no longer re-examine the tax return 
and issue a reassessment. Normally this date 
occurs three or four years after the initial 
assessment. 
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Court case response—restrictive covenants 


When a taxpayer sells a business, part of the selling price may be linked to an 
agreement by the seller to not compete with the business carried on by the buyer. This 
kind of agreement is called a non-compete agreement, or restrictive covenant. The 
common understanding of tax policy and its interpretation by the Canada Revenue 
Agency was that amounts received by a vendor in respect of a restrictive covenant on 
the sale of shares of a corporation were taxable. However, court decisions in 2000 and 
2003 ruled that these amounts were generally not taxable.’ 


In response to the court decisions, the Minister of Finance issued a news release 

in 2003 announcing his intention to propose amendments to the /ncome Tax Act that 
would generally treat amounts received in this way as ordinary income for tax 
purposes (or in some important exceptions, as a capital gain). As with the case study 
on the routine technical change, proposed amendments were tabled in Parliament in 
bills C-33 and C-10, but have not been enacted. 


Currently, taxpayers can report such amounts for tax purposes based on the court 
decisions or based on the draft legislation. If a taxpayer chooses to report based on 
the court decisions and the tax year becomes statute-barred before the new rules are 
enacted, the Agency would not be able to reassess the returns. This would result in 

a loss of tax revenue for the government. 


Source: OAG audit files 


The Department has a backlog of necessary technical amendments 


3.31 The list of outstanding technical amendments to the Income Tax 
Act has been growing. In addition to the 155 clauses of technical 
amendments included in Bill C-10, we estimate that there are at least 
250 technical amendments identified by the Department that have not 
yet been drafted or released for comment. 


3.32 The Department cannot fix this problem alone. However, in the 
past, it has released packages of proposed technical amendments to the 
public for comment. This practice informed taxpayers and tax 
practitioners about proposed changes even though they were only 
included in a bill some time later. By releasing draft amendments for 
comment, taxpayers are alerted to proposed changes and any concerns 
can be considered before the bill is introduced in the House of 
Commons. Over the past two years, apart from the draft legislation 
needed to implement technical changes identified in the previous 
budget, the Department has not released for comment any draft 
clarifying legislative proposals. Department of Finance Canada officials 
told us their practice was to release new draft proposals for comment 
only after the current technical bill had received royal assent. 


3.33 Department officials have also told us that undertaking the work 
to develop and draft technical amendments to remedy identified 
technical legislative deficiencies is not the Tax Legislation Division's 
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only priority. They indicated that their main priorities are giving policy 
advice to the Minister and developing annual budget implementation 
legislation. 


3.34. A recommendation concerning the current backlog of technical 
amendments appears at the end of this section. 


Priority issues in comfort letters are not being legislated on a timely basis 


3.35 Comfort letters issued to taxpayers do not bind the government. 
A comfort letter clearly states that the Department will recommend to 
the Minister that an amendment be made, but the actual amendment 
to income tax law is a matter for Parliament. The topic of comfort 
letters was discussed during various hearings of the Senate Standing 
Committee on Banking, Trade and Commerce in 2007 and 2008 as 
part of Parliament’s review of Bill C-10. During these hearings, the 
Senate was told that although comfort letters do provide clarity and a 
level of assurance to taxpayers and tax practitioners, they are not law 
and, therefore, still require an amendment to the Income Tax Act. 


3.36 During our interviews with tax practitioners across the country, 
we sought their views on the comfort letter process. Most of them were 
pleased that the letters were issued in a timely manner, and they 
thought that these letters were essential to facilitating economic 
activity and were an integral part of how Canada’s tax system functions. 
However, the practitioners expressed a need for the legislative changes 
that the comfort letters identified to be enacted. Currently, the 
accounting rules for preparing a public corporation’s financial 
statements require them to be based on tax legislation that has been 
substantively enacted. Consequently, the tax results of a comfort letter 
cannot be used in preparing audited financial statements. 


3.37 Between 1996 and 2008, the Department of Finance Canada 
issued 297 comfort letters. Only 46 of the issues dealt with in these 
comfort letters have become law. Some of these identified issues 
became amendments and were included in Bill C-10, which did not 
become law; other amendments have not yet been drafted. According 
to Department officials, legislation has not been developed or released 
for most comfort letters issued after 2002. Thus there are as many as 
250 identified legislative deficiencies that require technical 
amendments from comfort letters alone. 


3.38 The following case study is an example of the uncertainty that 
taxpayers face when an income tax return has been filed based on a 
comfort letter. 


Report of the Auditor General of Canada—Fall 2009 


INCOME TAX LEGISLATION 


The taxation of dividends was dealt with in a comfort letter but not passed into law 


A corporation that pays dividends on certain preferred shares has to pay tax under Part 
VI.1 of the Income Tax Act, but it can claim a deduction from income to approximate 
this tax paid. Until a legislative change is made, the cost of financing a corporation’s 
operations with preferred shares has increased. This is because a corporation will 
typically pay tax under Part VI.1 at a rate of 40 percent on the amount of dividends 
paid. It is then entitled to a deduction equal to three times the amount of Part VI.1 tax 
paid. When the combined federal and provincial corporate tax rate has been 33.33 
percent, the tax saved from the deduction has equalled the Part VI.1 tax paid. Because 
the combined tax rate has been reduced in recent years, the tax saved from the 
deduction is no longer sufficient to offset the Part VI.1 tax paid. This would, if not 
corrected, result in increased financing costs that were not anticipated when the 
preferred shares were issued. 


The Department of Finance Canada issued a comfort letter in 2002 stating that it 
would recommend that changes be made to the deduction. The changes were part of 
Bill C-33 and Bill C-10 (described in Exhibit 3.2) and remain outstanding some seven 
years later. 


Source: OAG audit files 


Taxpayers are assessed based on comfort letters 


3.39 The Department issues about 30 comfort letters each year. These 
are released under Access to Information requests and published by 
private sector tax information services. Other taxpayers may use these 
comfort letters to calculate their tax liabilities. The Agency has no 
processes to identify and monitor the taxpayers who have carried out 
transactions based on comfort letters. This creates a risk that tax 
returns that have been completed based on a comfort letter may 
become statute-barred before the legislative change described in the 
comfort letter is enacted by Parliament. The Agency does not manage 
this risk by obtaining waivers from the taxpayers, which would allow it 
to later reassess the tax return based on the existing law, should the 
legislative change outlined in the comfort letter not be made. 
Although we are not aware of any situation when Parliament refused 
to enact a proposed change to the Act that had arisen from a comfort 
letter, we are concerned that taxpayers are assessed based on letters, 
the draft legislation for which may not be tabled and passed for years. 


3.40 Recommendation. The Department of Finance Canada should 


¢ develop and implement a plan to address the current backlog of 
necessary technical amendments; and 


¢ regularly develop and release draft technical amendments, 
including those that arise from comfort letters, so that taxpayers 
and tax practitioners know what changes will be made and can 
provide input to the Department. 
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The Agency’s legislative 
maintenance input 


Chapter 3 


The Department’s response. Agreed. Tax legislation is prepared by 
the Department and then tabled in Parliament by the Government. 
Recent practice with respect to legislative proposals to implement 
technical amendments (commonly known as tax “technical bills”) has 
been to await Parliament’s consideration and adoption of one technical 
bill before proceeding to release for public comment another package 
of proposed draft technical legislative amendments. This procedure 
has minimized potential confusion for taxpayers and their advisors (for 
example, by having different amendments to the same section of the 
Income Tax Act before Parliament and released in draft form for 
comment) and, in an environment in which draft technical bills have 
been considered and adopted by Parliament within a reasonably short 
period following their original release in draft form, has not resulted in 
any inordinate delay in the preparation and release of subsequent 
technical amendment packages. 


Where proposed technical legislative amendments are not enacted 
within a reasonably short period following their original release in draft 
form, the Department recognizes that the current practice of seeking 
to defer the release of a new package of the draft technical 
amendments may raise different issues relating to taxpayer uncertainty. 
To reflect these competing factors, the Department agrees to consider 
whether there are circumstances where it would be appropriate to 
bring forward for consideration a subsequent draft technical 
amendments package, notwithstanding the fact that the previous 
technical bill had not yet been adopted by Parliament. 


The Agency does not have a comprehensive system to track, record, and prioritize 
legislative issues 


3.41. We examined whether the Canada Revenue Agency maintains a 
complete list of legislative issues that it has identified as needing 
technical amendments, and whether it communicates these to the 
Department of Finance Canada. We found that the Agency uses 
several methods to identify technical issues, but it does not refer all 
issues to the Department and does not keep a consolidated inventory. 


3.42 The Agency’s Legislative Policy Directorate of the Legislative 
Policy and Regulatory Affairs Branch acts as the link between other 
branches of the Agency and the Department for identifying tax 
legislative issues. The Directorate receives information about 
legislative issues from Agency branches in several ways and 
communicates regularly with the Department. Occasionally, other 
areas of the Agency will contact the Department directly, but they 
need to inform the Directorate of any technical issues. 
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3.43 Referrals from the Agency. Various branches within the Agency 
identify legislative issues and communicate these to the Directorate. 
Staff in the Directorate ensure that the issues are valid and analyze 
some of them. They do not systematically review what impact these 
issues might have on compliance or on the tax base. The Directorate 
will communicate issues that it thinks need legislative amendments to 
the Department when they are identified; last year, it sent about 

20 letters to the Department. 


3.44 The Directorate exchanges many emails with the various 
branches in connection to their ongoing discussions of legislative 
issues—about 26,000 in the past four years. It is difficult to assess how 
many technical issues are described in these emails as they are not 
catalogued by the Directorate. 


3.45 Agency input into the budget process. Another approach that 
the Legislative Policy Directorate uses to gather and communicate 
details on important issues identified by Agency staff is to request a list 
of priority issues from the Agency’s branches before the annual federal 
budget is developed. Although the budget generally deals with tax 
policy issues, occasionally a technical issue will be included in the 
budget. In advance of the 2008 Budget, the Directorate compiled a list 


of 47 suggestions for changes to the legislation from within the Agency. 


From these 47 suggestions, it sent 10 to the Department. Six of these 
ten issues were addressed in the 2008 budget; two of the six that were 
addressed were technical. The issues that the Agency forwards to the 
Department before the Budget is introduced are chosen based on an 
assessment and prioritization by senior Agency officials. The Agency’s 
Policy Committee, chaired by the Assistant Commissioner of the 
Legislative Policy and Regulatory Affairs Branch, makes this 
assessment. Agency officials told us that they do not necessarily refer 
all technical issues that they think need to be addressed by a legislative 
amendment to the Department. Instead, they choose the ones that 
they think will fit the theme of the Budget. The same changes 
considered to be priorities may be referred to the Department year 
after year if necessary. 


3.46 Agency input to technical bills. When the Directorate is 
informed by the Department that a technical bill is being prepared, it 
prepares a list of Agency priorities for legislative change to send to the 
Department. As there is no consolidated list of issues, the Directorate 
contacts various Agency branches to ask for their list of outstanding 
issues, and checks its own files. The current list of income tax 
legislative issues identified by the Compliance Programs Branch 
includes approximately 150 items. The lists of issues submitted to the 
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Department (for possible inclusion in a technical bill) in 2008 and 

in 2005 contained 108 and 54 items, respectively. The following case 
study is an example of an important technical issue that was identified 
by the Agency. 


Canada Revenue Agency identifies a potential legislative issue 


This is an example of the Canada Revenue Agency identifying a potential legislative 
_issue and communicating it to the Department of Finance Canada. 


The /ncome Tax Act contains many provisions that are designed to ensure that a 
corporation’s tax losses cannot be used by unaffiliated corporations unless certain 
conditions are met. 


In recent years, some transactions have been designed to get around these rules, 
described as “tech-wreck” restructurings. The name arose from the use of losses 
originating in the technology sector. 


The Agency’s Legislative Policy Directorate wrote to the Department on 1 February 
2001, following up on a request from the Income Tax Rulings Directorate about 
inappropriate use of losses. 


The Agency’s Legislative Policy Directorate wrote to the Department on 8 March 2004 
regarding more recent inappropriate corporate reorganizations involving losses. 


A meeting was held in August 2005 on the trading of non-capital losses and other 
losses. Staff from the Agency and the Department attended. 


The Agency’s Legislative Policy Directorate wrote to the Department again in 2007 on 
the same issues involving loss trading and informed the Department of another 
scheme that it believed was resulting in tax revenue losses. The Agency again asked 
the Department to recommend legislative amendments. 


To date, the issue remains unresolved. 


The Agency has stated in Income Tax Technical News #34 that it will apply the 
General Anti-Avoidance Rule to such transactions. Any resulting litigation will likely 
take several years before the uncertainty is resolved. 


Source: OAG audit files 


3.47 Overall, technical issues identified within the Agency are not 
systematically tracked and monitored and they are not formally 
documented by managers. Directorate files on legislative issues are not 
electronic, and it does not keep a correspondence log related to these 
issues. At present, only seven staff in the Directorate deal with changes 
to income tax legislation, so without an electronic record, the loss of 
one or two senior people could have a major impact on how effectively 
the Agency identifies and communicates the accumulated technical 
issues to the Department. 


3.48 Recommendation. The Canada Revenue Agency should create 
an electronic database to help validate, analyze, and prioritize 
identified technical issues that should be referred to the Department of 
Finance Canada and to track how they are addressed. 
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The Agency’s response. Agreed. An electronic database is being 
developed and will be in place by the end of 2009. Further, a strategy is 
being developed to enhance the Agency’s legislative recommendation 
process. | 


The Agency is not meeting its own time targets for advance income tax rulings for 
taxpayers 


3.49 The Canada Revenue Agency provides guidance to taxpayers to 
enable them to self-assess and comply with the law. Some of this 
guidance is more complex and is intended for users who need more 
detailed information. If the Agency’s guidance is not timely or correct, 
taxpayers may inadvertently fail to comply with the law or they may 
become frustrated because the information they need is not available. 
Either may lead to a loss of tax revenue or an overpayment that later 
must be adjusted. We looked specifically at the guidance provided when 
the interpretation and application of the legislation may be unclear. 


3.50 “The Income Tax Rulings Directorate is part of the Agency’s 
Legislative Policy and Regulatory Affairs Branch. Its mandate is to 
interpret Canada’s income tax law and explain how Canada’s tax law 
applies to given situations. This includes providing advance income 
tax rulings for a fee and technical interpretations of income tax law to 
taxpayers (either directly or through tax auditors who are dealing with 
taxpayers). It produces Income Tax Interpretation Bulletins and the 
Income Tax Technical News, which describe the Agency’s 
interpretation of income tax legislation. The Directorate also 
participates in tax conferences where it provides information about its 
interpretive positions. The tax practitioners we interviewed generally 
find the guidance provided by the Directorate to be helpful. 


3.51 An advance income tax ruling provides certainty to a taxpayer 
who receives it by guaranteeing the income tax effect of certain 
transactions the taxpayer plans to undertake. The rulings involve some 
of Canada’s most complex tax issues and the tax revenue implications 
can be significant. 


3.52 Each ruling outlines how the Agency will interpret certain 
provisions of Canada’s income tax law. These rulings are not published 
by the Agency but are released by the Agency under Access to 
Information provisions, with confidential details removed. The 
Agency’s position on certain technical matters developed in a ruling 
may be of interest to other taxpayers. The rulings are available to those 
who subscribe to private sector tax information services. 


Chapter 3 17 


INCOME TAX LEGISLATION 


18 Chapter 3 


3.53 The Directorate also provides technical interpretations when 
asked to do so. These interpretations give guidance in a hypothetical 
situation and are also released under Access to Information provisions. 
The Agency provides about 250 advance income tax rulings and 
1,200 technical interpretations to taxpayers and 1,000 technical 
interpretations to its auditors each year. 


3.54 We examined whether the Agency had well-managed processes 
in place to provide timely guidance to taxpayers on the interpretation 
and application of income tax provisions to enable them to self-assess 
and comply with the law. 


3.55 Generally, tax practitioners we interviewed told us that they value 
the Agency’s advance income tax rulings service because it provides 
them with certainty about clients’ proposed business transactions. 


3.56 They also told us that they find the response times from the 
Income Tax Rulings Directorate for advance tax rulings and technical 
interpretations to be slow. Some said that they no longer request 
technical interpretations and advance income tax rulings because they 
find that the response time is too long to enable them to meet their 
clients’ needs. 


3.57 The Agency has met its service standard target of 90 days for 
income tax technical interpretations in five of the last six years. 
However, this standard is based on “controllable time,” which includes 
only the time when the file is being worked on by the directorate or 
any other government department. This may explain the practitioners’ 
perception of a lack of timeliness. 


3.58 The Agency has recognized in its Annual Report to Parliament 
that it has not met its own service standards for advance income tax 
rulings and acknowledged that this may result in taxpayers 
undertaking questionable transactions—potentially leading to reduced 
compliance and more costly audit and appeals activities (Exhibit 3.3). 


3.59 Agency officials and tax practitioners told us that one reason for 
the lack of timeliness is that the requested rulings are becoming more 
and more complex. Very few involve simple transactions that can be. 
dealt with quickly; the Agency recognizes the tax risk associated with 
the proposed transactions. Moreover, 61 percent of the current staff in 
the Income Tax Rulings Directorate was not employed in 2004, so they 
are still gaining experience. The Agency’s branch business plan 
recognizes the challenge, but does not have a concrete plan to improve 
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its performance in the short term. Management believes that the 
addition of 16 new staff in the past year, once they gain more 
experience, will help with timeliness. 


Exhibit 3.3 In recent years, the Canada Revenue Agency has not met its own targets for issuing advance income tax rulings 


Advance income tax rulings 


© _ Advance income So "Income tax technical interpretations 
Target is 60 days (of time within the government's control) Target is 90 days (of time within the government's control) 
Fiscal year Average number of days Rating standard: Average number of days Rating standard: 
to issue to taxpayers Met or Not Met to issue to taxpayers Met or Not Met” 
Aan ae Le ae, Ge eS ee nnn 
2002-03 60 days Met 78 days Met 
2003-04 57 days Met 82 days Met 
2004-05 62 days Met 75 days Met 
2005-06 84 days Not Met 87 days Met 
_ 2006-07 94 days Not Met 105 days Not Met 
2007-08 101 days Not Met 89 days Met 


Source: Canada Revenue Agency’s Annual Report to Parliament 2007-08, and prior years 
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3.60 Recommendation. The Canada Revenue Agency should 
develop more concrete plans to meet its own target times for issuing 
advance income tax rulings, given the significance of the rulings to 
proposed business transactions. 


The Agency’s response. Agreed. The Agency will formally evaluate 
these performance standards to ensure they adequately reflect the time 
needed for issuing advance rulings, given the increased complexity of 
these cases. This study of turnaround times and trends will be 
completed in 2010. 


It is not known whether the guidance provided to tax auditors is timely 


3.61 We examined whether the Agency had a well-managed process 
in place to provide timely guidance to Agency auditors across the 
country on the interpretation and application of income tax law. We 
found established processes to provide the guidance needed, but 
management does not have any reliable information on whether or not 


it is timely. 
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3.62 The Agency has established an internal, 60-day standard for 
responses to queries from auditors of small and medium businesses, 
although there is no standard response time established for queries 
from large corporations. However, the Agency does not record or track 
the actual response time to any of these queries. As a result, the 
Agency does not know whether the guidance it provides to its own 
auditors is timely. 


Users are not always aware that some Income Tax Interpretation Bulletins are 
obsolete 


3.63 Income Tax Interpretation Bulletins are produced by the Income 
Tax Rulings Directorate. They provide the Agency’s technical 
interpretation and position on certain provisions of income tax law. For 
the majority of taxpayers, the explanations of income tax law available 
in the Agency’s other publications, such as tax guides and pamphlets, 
are sufficient. 


3.64 The Income Tax Technical News is also produced by the Income 
Tax Rulings Directorate and gives the Agency’s technical 
interpretation of various provisions of income tax law and informs 
users of a new or revised position. The Income Tax Technical News 
began in 1994 and, to date, 40 issues have been published. 


3.65 Information Circulars provide information to the general public 
on administrative, enforcement, or procedural matters that relate to 
income tax law. 


3.66 Most tax auditors and many tax practitioners told us that they 
begin their research into a legislative interpretation question with 
Income Tax Interpretation Bulletins as their source of guidance. As of 
31 December 2008, there were 272 bulletins in effect. Tax auditors and 
tax practitioners have told us that, although they are aware that some 
of the guidance in the bulletins may not be up to date, they continue to 
rely on them to support their positions when they are uncertain about 
how to interpret income tax law. Agency statistics indicate the 
importance of the bulletins, using as evidence the 1.5 million page 
views of the bulletins on its website for the fiscal year ended 

31 March 2009. These views were from users external to the Agency 
and do not include Agency staff. Users may also access the bulletins 
through their subscription to private sector tax information services. 


3.67. Although the Income Tax Rulings Directorate’s Policies and | 
Procedures Manual contains a clear process for developing and 
producing an Income Tax Interpretation Bulletin, we found that there 
is no formal process to monitor whether it is still current or to decide 
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whether it needs to be withdrawn or revised. Since 2003, only 29 of 
the existing bulletins have been revised and only one new one was 
issued. The Agency told us that the production and revision of 
bulletins is not a priority, in part due to resource constraints. 


3.68 We expected that if all or part of an Income Tax Interpretation 
Bulletin was no longer correct, the Agency would communicate this to 
users by updating the bulletin. We found that there is no indication in 
the bulletin itself that specific paragraphs may no longer be current; 
there is only the general comment in the index listing of all bulletins 
that the information in certain ones may not be up to date. The Agency 
does provide information about new interpretive positions in its Income 
Tax Technical News publication. However, this approach requires a user 
to know that a specific bulletin may be out of date and to read all issues 
of Income Tax Technical News to see if anything has been said about 
the one they plan to rely on. The Agency may issue a special release 

of the bulletin to communicate changes. However, the user may be 
required to do further research, such as looking at changes to the 
legislation, conference presentations, current articles, and recent court 
cases to establish the correct interpretive position. The following case 
study is an example of the effect of an out-of-date bulletin. 


Example of an Interpretation Bulletin that is out-of-date 


A scholarship, fellowship, bursary, prize, research grant, and other kind of financial 
assistance for scholars is included in income in the year received, subject to a 
“scholarship exemption.” The scholarship exemption may result in a full deduction 
from income, resulting in no tax owing. The Agency issued Income Tax Interpretation 
Bulletin IT-75R4 on 18 June 2003. This IT discusses the taxation of the scholarships 
and similar prizes. It indicates the scholarship exemption exempts the first $500 and 
in some cases the exemption is $3,000. 


In fact, the scholarship exemption changed for taxation years after 2005. The full 
amount may qualify for the exemption, and therefore not be subject to tax. IT-75R4 
does not reflect the legislative change and can mislead a user into reporting the wrong 
amount on the tax return. 


Source: OAG audit files 


3.69 Recommendation. The Canada Revenue Agency should 
improve the information it provides to users about specific paragraphs 
in Income Tax Interpretation Bulletins that are no longer accurate, so 
that they can more readily comply with the Income Tax Act. 


The Agency’s response. Agreed. The Agency provides taxpayers and 
tax practitioners with up-to-date technical information such as 
pamphlets, guides, tax return packages, technical opinions and 
conference responses. Given that there are several other mediums that 
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provide this type of technical information, the Agency may opt to 
cancel its inventory of Interpretation Bulletins. To that end, the 
Agency will evaluate this option during 2010 and ensure practitioners 
are consulted and aware of any subsequent plan of action to either 
update these bulletins or cancel them. 


Conclusion 


3.70 Some aspects of maintaining the clarity of the Income Tax Act are 
not being well managed, which creates difficulties for taxpayers and 
administrative complications for the Canada Revenue Agency. 
Although the government has agreed a number of times to introduce 
annual technical bills to correct identified deficiencies, it has not done 
so. The last technical bill on income tax law received royal assent 

in 2001. Each year, more deficiencies are identified, contributing to an 
ever-growing backlog of needed technical amendments. 


3.71 The Department of Finance Canada alone cannot correct this 
situation, but it can do more to bring the urgency of the problem to the 
attention of the government and Parliament and make known, by 
releasing draft amendments for stakeholder comment, how the 
important technical deficiencies will be remedied. 


3.72 Although the Agency does collect information from its own 
operations about technical issues in tax legislation, it records them in 
various ways, does little to analyze their impact, and does not 
communicate all of the identified issues requiring amendments to the 
Department of Finance Canada. The Department has weaknesses in 
its own process for recording and monitoring legislative issues. 


3.73 The Agency’s guidance to taxpayers and its own tax auditors, 
through advance income tax rulings and technical interpretations, has 
well-managed aspects, but the advance income tax rulings are not 
being provided in a timely manner. Income Tax Interpretation 
Bulletins are frequently referred to by taxpayers and tax practitioners, 
but are not being kept up to date. Revising these bulletins is not a 
priority for the Agency, and the public is not always made aware of the 
fact that a bulletin may be obsolete. 
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About the Audit 


All of the audit work in this chapter was conducted in accordance with the standards for assurance 
engagements set by The Canadian Institute of Chartered Accountants. While the Office adopts these 
standards as the minimum requirement for our audits, we also draw upon the standards and practices of 
other disciplines. 


Objectives 
The objective of the audit was to determine whether 


- the Canada Revenue Agency has well-managed processes in place to help taxpayers comply with the 
Income Tax Act, by providing timely guidance to clarify the interpretation and application of the 
legislation; and 


- the Agency and the Department of Finance Canada have well-managed processes in place to identify 
and develop technical amendments to the income tax legislation considered necessary to more clearly 
reflect the policy intent, to facilitate compliance, and to protect the tax base. 


Scope and approach 


Our audit focused on two areas of administration of the Income Tax Act: the activities that respond to 
compliance problems related to taxpayer uncertainty arising from the application and interpretation of the 
income tax legislation and the processes in place to develop the necessary technical legislative changes in 
a timely manner. We focused our work in specific sectors of the Agency and the Department, with our 
main focus being directed at the Agency’s Legislative Policy and Regulatory Affairs Branch and the 
Department’s Tax Policy Branch. 


During our audit, we interviewed tax services office auditors and reviewed documents they gave us. We 
then interviewed headquarters staff to understand how the Agency ensures that it is providing tax auditors 
and taxpayers with consistent guidance on how to apply and interpret tax legislation. 


We interviewed 54 tax practitioners from across the country. Each was asked a series of questions 
developed by the audit team. The tax practitioners were referred to us by the Canadian Tax Foundation 
and reflected a target population of senior tax practitioners (both accountants and lawyers) with practices 
in larger Canadian cities and with client bases composed of both large and small clients. We wanted to 
hear the views of a target population that would be most likely to use the services of both the Canada 
Revenue Agency and the Department of Finance Canada in the course of their work. We conducted these 
interviews over a 19-month period in Vancouver, Calgary, Toronto, and Montreal. 


Report of the Auditor General of Canada—Fall 2009 Chapter 3 | 23 


INCOME TAX LEGISLATION 


Criteria 


Listed below are the criteria that were used to conduct this audit and their sources. 


 Oiteia 


We expected that the Canada Revenue Agency had well- 
managed processes in place to provide timely guidance to 
Agency officials and taxpayers on the interpretation and 
application of income tax provisions through advance income tax 
rulings, technical interpretations, information circulars, income 
tax interpretation bulletins, and the Income Tax Technical News. 


Canada Revenue Agency, 2008-09 to 


2010-11 Legislative 
Policy and Regulatory Affairs Branch Business Plan, pages 7, 
13, 14, and 36 

Canada Revenue Agency, 2008-09 Report on Plans and 
Priorities, Appendix C, page C-2 


We expected that the Department of Finance Canada had well- 
managed processes in place to identify, analyze, prioritize, and 
draft technical amendments to ensure the effective functioning of 
Canada’s tax system. 


Department of Finance Canada, 2007-08 Report on Plans 
and Priorities, page 18 


We expected that the Agency had well-managed processes in 
place to monitor taxpayers who have been assessed based on 
comfort letters. 


Canada Revenue Agency, 2006-07 Annual Report to 
Parliament, page 9 


We expected that the Agency had well-managed processes in 
place to identify, analyze, and prioritize amendments needed to 
address the lack of clarity in the /ncome Tax Act and to 
communicate these to the Department of Finance Canada in a 
timely manner. 


Treasury Board of Canada Secretariat, Risk Management 
Policy, Preface, 

Canada Revenue Agency, 2008-09 to 2010-11 Legislative 
Policy and Regulatory Affairs Branch Business Plan, page 7 
Canada Revenue Agency, 2006-07 Report on Plans and 
Priorities, page 45 


Management reviewed and accepted the suitability of the criteria used in the audit. 


Period covered by the audit 


~ 


The period under audit is from 2004 to 2009. Other historical documents reviewed were pertinent to the 


time period under audit and were published as far back as 1973. 


Audit work for this chapter was substantially completed on 24 June 2009. 


Audit team 
Assistant Auditor General: John Rossetti 
Principal: Vicki Plant 


Lead Director: Brenda Siegel 
Director: Heather Miller 


Sophie Chen 
Natanz Handy 
Suzanne Moorhead 


Rod Newcombe 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 3. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


The Department’s legislative maintenance 


3.27. The Department of Finance 
Canada should use an integrated 

and consistent process for recording, 
tracking, and prioritizing all technical 
issues for possible legislative 
amendment. (3.18-3.26) 


3.40 The Department of Finance 
Canada should 


¢ develop and implement a plan to 
address the current backlog of 
necessary technical amendments; and 


* regularly develop and release draft 
technical amendments, including 
those that arise from comfort letters, 
so that taxpayers and tax 
practitioners know what changes 


will be made and can provide input 
to the Department. (3.28-3.39) 
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Agreed. While systems are in place to manage and keep track 
of technical issues for possible legislative amendment, 
improvements can be made with regard to database management 
and usage. The Department will undertake, as resources permit, 
to consolidate its system for ensuring that technical issues are 
documented and catalogued consistently and that this system 

is maintained and kept up to date as new issues arise, and as 
existing issues are addressed. 


Agreed. Tax legislation is prepared by the Department and then 
tabled in Parliament by the Government. Recent practice with 
respect to legislative proposals to implement technical amendments 
(commonly known as tax “technical bills”) has been to await 
Parliament’s consideration and adoption of one technical bill before 
proceeding to release for public comment another package of 
proposed draft technical legislative amendments. This procedure 
has minimized potential confusion for taxpayers and their advisors 
(for example, by having different amendments to the same section 
of the Income Tax Act before Parliament and released in draft form 
for comment) and, in an environment in which draft technical bills 
have been considered and adopted by Parliament within a 
reasonably short period following their original release in draft form, 
has not resulted in any inordinate delay in the preparation and 
release of subsequent technical amendment packages. 


Where proposed technical legislative amendments are not enacted 
within a reasonably short period following their original release in 
draft form, the Department recognizes that the current practice of 
seeking to defer the release of a new package of the draft technical 
amendments may raise different issues relating to taxpayer 
uncertainty. To reflect these competing factors, the Department 
agrees to consider whether there are circumstances where it would 
be appropriate to bring forward for consideration a subsequent draft 
technical amendments package, notwithstanding the fact that the 
previous technical bill had not yet been adopted by Parliament. 
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The Agency’s legislative maintenance input 


26 


3.48 The Canada Revenue Agency 
should create an electronic database to 
help validate, analyze, and prioritize 
identified technical issues that should 
be referred to the Department of 
Finance Canada and to track how they 


are addressed. (3.41-3.47) 


The Agency’s legislative guidance 


3.60 The Canada Revenue Agency 
should develop more concrete plans to 
meet its own target times for issuing 
advance income tax rulings, given the 
significance of the rulings to proposed 
business transactions. (3.49-3.59) 


3.69 The Canada Revenue Agency 
should improve the information it 
provides to users about specific 
paragraphs in Income Tax 
Interpretation Bulletins that are no 
longer accurate, so that they can more 


readily comply with the Income Tax Act. 


(3.61-3.68) 
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Agreed. An electronic database is being developed and will be in 
place by the end of 2009. Further, a strategy is being developed 
to enhance the Agency’s legislative recommendation process. 


Agreed. The Agency will formally evaluate these performance 
standards to ensure they adequately reflect the time needed for 
issuing advance rulings, given the increased complexity of these 
cases. This study of turnaround times and trends will be 
completed in 2010. - 


Agreed. The Agency provides taxpayers and tax practitioners 
with up-to-date technical information such as pamphlets, 
guides, tax return packages, technical opinions and conference 
responses. Given that there are several other mediums that 
provide this type of technical information, the Agency may opt 
to cancel its inventory of Interpretation Bulletins. To that end, 
the Agency will evaluate this option during 2010 and ensure 
practitioners are consulted and aware of any subsequent plan of 
action to either update these bulletins or cancel them. 
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What we examined 


Electronic Health Records 


Main Points 


Canada Health Infoway Inc. (Infoway) was created in 2001 as a 
federally funded, not-for-profit corporation to lead the national 
development of electronic health records (EHRs). Infoway’s goal is to 
ensure that, by 2010, every province and territory and the populations 
they serve will benefit from new health information systems that will 
help transform their health care system. Furthermore, Infoway’s stated 
goal is that 50 percent of Canadians will have their electronic health 
record available to their authorized health care professionals by 2010, 


and 100 percent by 2016. 


Health Canada transfers funds to Infoway based on funding 
agreements that specify the obligations of each party. Infoway has been 
granted $1.6 billion by the federal government. Infoway has allocated 
$1.2 billion to invest jointly with the provinces and territories in 
projects to accelerate the development and use of compatible EHRs 
across Canada, and $400 million for other related priorities. It is the 
role of the provinces and territories to propose the projects to Infoway 
for funding and to implement them. Thus, Infoway’s ability to achieve 
key outcomes in its funding agreements depends on the collaboration 
of the provinces and territories. 


We examined how Infoway manages funds from the federal 
government to achieve its goal of making compatible electronic health 
records available across Canada. In addition, we looked at the role of 
Health Canada, the sponsoring department, in ensuring that Infoway 
complies with the agreements under which it receives funding from the 
Department. For the most part, the audit focused on the fiscal 

years 2006-07 and 2007-08. Audit work for this chapter was 
substantially completed on 30 April 2009. 


Concurrent with our audit, six provincial audit offices audited how 
electronic health records funded by Infoway and/or provincial 
governments are being implemented in their respective provinces. The 
provincial audit offices will each report separately; a joint summary 
report on all of the audits will be issued in 2010. 
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ELECTRONIC HEALTH RECORDS 


Why it’s important 


What we found 


2 Chapter 4 


Electronic health records (EHRs) are intended to offer solutions to a 
number of persistent problems in Canada’s health system, some of 
which can be attributed to the use of paper-based health records. With 
EHRs, it is expected that health care professionals would be better able 
to share patient information, thus avoiding unnecessary or duplicate 
diagnostic tests, multiple prescriptions, and the risk of adverse drug 
reactions. Ultimately, the use of EHRs could reduce patient wait times, 
reduce costs, and save lives. From its inception in 2001 until 

31 March 2009, Infoway had spent $614.9 million on this initiative, 
and had committed an additional $614.2 million, for a total of 

$1.2 billion. Some experts have estimated the total cost of 
implementing EHRs Canada-wide at over $10 billion, and Infoway 
concurs. 


« Infoway has accomplished much in the eight years since its creation. 
Using the funding agreements with Health Canada as a starting 
point, Infoway developed an approach to providing for compatible 
electronic health records by identifying the key requirements and 
components of an EHR and developing a blueprint for the design of 
health information systems. It consulted widely with partners and 
stakeholders to obtain their input and support. In addition, it 
established appropriate governance mechanisms and developed a 
risk management strategy. It has implemented appropriate 
management controls for operational spending, although controls for 
contracting for goods and services need to be strengthened. 


¢ In the 29 EHR projects we examined, Infoway had ensured that 
provinces and territories designed the projects to comply with 
requirements such as its blueprint and standards. It had also 
identified project-specific risks and was monitoring them, as well as 
other problems that arose during the life of the project. However, 
Infoway has not obtained.the results of conformance testing on EHR 
systems. This means it does not have sufficient assurance that 
standards have been implemented as required. 


¢ Infoway has made considerable efforts to report on the progress of 
the EHR initiative. It reports progress toward its 2010 goal as the 
percentage of Canadians living in provinces or territories where an 
EHR is available to their health care professionals. However, it has 
not reported on other indicators of progress, such as the extent to 
which completed systems meet requirements for compatibility. Nor 
does it report on the adoption or use of completed systems by health 
care professionals, although it considers low adoption rates a serious 
risk to the EHR initiative. 


Report of the Auditor General of Canada—Fall 2009 


ELECTRONIC HEALTH RECORDS 


- As the sponsoring department, Health Canada periodically obtains 
assurance through audits and evaluations that Infoway is complying 
with the funding agreements. However, at the time of the audit, the 
Department still had not fully developed the monitoring framework 
it approved in 2008 to manage risks associated with such large 
amounts of funding and to strengthen ongoing monitoring of the 
Corporation. 


The entities have responded. The entities agree with all of our 
recommendations. Their detailed responses follow the 
recommendations throughout the chapter. 
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ELECTRONIC HEALTH RECORDS 


Introduction 


4.1 Each year in Canada, hundreds of millions of health care-related 
transactions occur. Almost all involve handwritten records. This 
critical information is filed in doctors’ offices and clinics, and in nearly 
1,000 hospitals. In September 2000, the first ministers unanimously 
agreed “to work together to strengthen a Canada-wide health 
infostructure to improve quality, access and timeliness of health care 
for Canadians” and committed to developing electronic health records. 


42 ‘To meet this commitment, Canada Health Infoway Inc. 
(Infoway) was created in 2001 as a not-for-profit corporation at 
arm’s-length from the government. It works with the provinces and 
territories to foster and accelerate the development and adoption of 
electronic health records across Canada. 


Definition of an electronic health record 


43 Anelectronic health record (EHR) is a secure and private 
lifetime record that describes a person’s health history and care. 
Such a record would be available electronically to authorized health 
care professionals anywhere and anytime to support the provision of 
high-quality care. 


44 The terms electronic health record and electronic medical 
record (EMR) are often interchanged. However, in the context of this 
audit, they must be clearly distinguished. An EHR refers to a person’s 
health record that can be accessed online from many separate, 
compatible systems within a network. An EMR refers to an electronic 
patient record that can be accessed from a single system in a doctor’s 
office and that may, or may not, be shared with other health care 
professionals. 


45 An EHR that is fully functional will allow health care 
professionals to view and update a patient’s health record. The EHR 
would include a person’s 


¢ age, address, and other personal details; 
¢ diagnostic imaging, drug, laboratory, hospital, and clinical reports; 
* infectious disease and immunization records; and 


© other health information. 
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4.6 


To implement an EHR, Infoway has determined that the core 


systems described in Exhibit 4.1 must be in place. 


Exhibit 4.1 Systems that support an electronic health record, including expenditures and commitments as of 31 March 2009 


Expenditures Commitments Total 


Name of system | ; | ($ millions) ($ millions) ($ millions) 


Registries 


Client Registry—A list of all patients and their relevant personal information, such as 


names and addresses. 


Provider Registry—A list of participating health care professionals who are 100.0 32.2 132.2 


authorized to use the system. 


When completed by all jurisdictions, these registries will uniquely identify all 


Canadians and their doctors. 


Diagnostic imaging System (DI) 


A system that electronically collects, stores, manages, distributes, and displays a 217.1 bees 3356 


patient's images and reports, such as X-rays, ultrasounds, MRIs, and CT scans. All 
this information would exist in digital format, without the need for film. 


~ Drug Information System (DIS) 


A system that allows health care professionals to access, manage, share, and 


safeguard a patient’s medication history. For each new prescription, the system will 101.4 140.1 241.5 


check for allergy alerts and drug-to-drug interactions against a complete medication 


profile. 


Laboratory Information System (LIS) 


No matter where a patient is tested, this system will allow laboratory technicians to 


enter results into a database. Test results will be linked to the patient’s EHR and will 71.8 92.5 164.3 


give health care professionals more information to aid in diagnosing and treating 


patients. 


Interoperable Electronic Health Record (iEHR) 


This system will allow authorized health care professionals to view and, in some 
cases, to update a patient’s essential health information. 


Interoperable refers to a system that has the ability to work with other systems or 
products. lf they weren't part of an interoperable electronic health record (iEHR), the 
registries, diagnostic imaging, drug information, and laboratory information systems 
would be unable to send or receive information from health care professionals. 


Total 


Source: Adapted from Infoway information 


4] 


124.6 220.9 300-0 


SSS ears ep eee rr emcee eerste ese aaa asec etme 
614:9 614.2 122g: 


In creating EHRs, thousands of systems will need to be 


integrated. As a result, the implementation of an EHR system is an 


expensive, long-term initiative. Since its creation in 2001 until 

31 March 2009, Infoway had spent $614.9 million on various EHR 
projects from the systems described in Exhibit 4.1 and had committed 
an additional $614.2 million, for a total of $1.2 billion. Some experts 
have estimated the total cost of implementing EHRs in Canada to be 
more than $10 billion. Infoway concurs with this estimate. 
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48 The development and implementation of EHRs is no simple 
matter. Canada’s health care system is huge. Millions of encounters 
between patients and health care professionals occur each day. 
According to Infoway, about 2,000 health care transactions happen per 
minute in Canada. Many are quite complex and all involve 
documentation and information flow. Each year there are 


¢ 440 million laboratory tests performed, 


382 million prescriptions filled, 


¢ 322 million visits to doctors’ offices, 


35 million diagnostic images taken, and 


2.8 million hospitalizations. 


49 Electronic health records are intended to provide solutions to a 
number of persistent problems in Canada’s health system, some of 
which may be caused by the use of paper-based health records. Records 
in electronic form are more likely to be legible, available when needed, 
and more easily and quickly retrieved and made available. Other 
potential benefits include the following: 


¢ For patients—improved health care and decreased risks (such as 
fewer adverse drug reactions); lower chance of having duplicate, 
invasive, or expensive tests; and shorter waiting lists. 


¢ For health care professionals—an integrated view of patient data 
and the ability to make better decisions, thanks to up-to-date 
patient information. 


¢ For health administrators—reduced health care costs and 
improved health care quality. 


¢ For governments—support for improved long-term planning and 
improved health resource allocation. 


In short, EHRs are expected to reduce costs, improve quality of care, 
and help save lives. 


Canada’s model for shared responsibility 


4.10 Infoway. Infoway describes its role as that of a “strategic 
investor” that makes focused investments to spur the development of 
EHRs across the country. It works with the provinces and territories, 
which are responsible for delivering health care, in setting a national 
direction, and it helps to ensure that provincial and territorial 
strategies are aligned with national priorities. 


Report of the Auditor General of Canada—Fall 2009 


Chapter 4 7 


ELECTRONIC HEALTH RECORDS 


Standards Collaborative—A body created 

in 2006 after extensive consultation with and 
approval by the boards of Infoway and the 
Canadian Institute for Health Information (CIHI), 
as well as the federal, provincial, and territorial 
Conference of Deputy Ministers of Health. Its 
mandate is to develop, maintain, and support 
the implementation of national standards. The 
Collaborative also plays a formal liaison role with 
organizations developing international 
standards. 


8 | Chapter 4 


4.11 Infoway funds projects with its provincial and territorial partners, 
paying 75 percent of eligible costs. It also participates in project 
planning, and monitors the progress of projects and the quality of 
deliverables by using a “gated funding model.” Provinces and territories 
are reimbursed only when specific milestones, or “gates,” have been 
achieved. Provinces and territories usually receive 20 percent of 
project funding when a legal agreement for a project is signed and 

30 percent of funding after Infoway has approved various project 
deliverables, such as project charters and system design documents. 


412 The most significant gate in Infoway’s funding approach is the 
“adoption gate,” which requires provinces and territories to show that 
certain core systems of an EHR have been adopted by health care 
professionals. Adoption criteria, which are the expected use of a core 
system by targeted health care professionals, are established for each 
project, such as a diagnostic imaging, laboratory, or drug information 
system. For example, adoption occurs when a hospital certifies that it 
no longer uses film for diagnostic imaging. When adoption criteria are 
achieved, Infoway reimburses the final 50 percent of the project’s 
approved budget. Linking 50 percent of project funding to the 
adoption of a core system provides some assurance to Infoway that 
EHR systems are beginning to be used, but it provides no assurance of 
widespread or ongoing use. 


4.13. Although each province and territory will have an EHR system 
adapted to its needs, it is important that provincial and territorial 
systems are based on an agreed set of principles and characteristics. To 
this end, Infoway collaborated with its partners on a design of health 
information systems for EHRs, called the Electronic Health Record 
Solution (EHRS) Blueprint (the Blueprint). Released by Infoway 

in 2003, the Blueprint provided a vision of how EHR information 
could be securely and appropriately shared across Canada using 
information and communications technologies. Revised in 2006, the 
Blueprint now provides more detail on how standards can support the 
sharing of health information and comply with federal, provincial, and 
territorial requirements, as well as privacy and security requirements 
across jurisdictions. Infoway believes that provincial and territorial 


‘alignment with the Blueprint and compliance with standards are 


essential to achieving EHRs that are compatible across the country. 


4.14 Infoway has also guided the development of standards through 
the Standards Collaborative. The Collaborative has developed 
standards for each core system of an EHR. This was done by consulting 
health care professionals, software vendors, and representatives from 
the provinces and territories. Standards for the registries, diagnostic 
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imaging, and drug information systems have been ready to use 
since 2006. Standards for laboratory information systems and 
interoperable EHRs have been available since 2007. 


415 Although Infoway takes steps to ensure that projects align with 
the privacy and security requirements outlined in the Blueprint, it has 
no responsibility for ensuring that systems comply with any privacy 
laws. 


4.16 Provinces and territories. The provinces and territories are 
responsible for developing their own EHR strategies and ensuring that 
they align with the Blueprint. Projects proposed by the provinces and 
territories and approved by Infoway are cost-shared, with Infoway 
paying 75 percent of eligible costs. Provinces and territories are 
responsible for implementing the projects and, because the total 
amount of funding to be provided by Infoway is capped when projects 
are approved, the provinces and territories assume all the risk of cost 
overruns. | 


4.17 The provinces and territories are responsible for the operating 
and maintenance costs of EHR systems. They also provide funds to 
regional health authorities and hospitals to cover other EHR initiatives 
not eligible for Infoway funding. The provinces and territories are 
responsible for ensuring that the implemented EHR systems comply 
with their own privacy laws. 


4.18 The provinces and territories are also responsible for helping 
doctors, nurses, pharmacists, and other health care professionals 
understand and use EHRs. This is critical in private offices or clinics, 
where few have shifted from paper-based patient records to electronic 
records. The success of implementing EHRs depends on the 
willingness of health care professionals to use the systems, and it is the 
responsibility of the provinces or territories to directly inform and gain 
support from them. 


4.19 Health Canada. Health Canada is the federal department that 
sponsors Infoway. It negotiated the funding agreements and prepared 
the related Treasury Board submissions. As the federal signatory to the 
agreements, Health Canada’s role is to ensure compliance with the 
funding agreements’ provisions. The Minister must account to _ 
Parliament for the effective ongoing administration of the agreements 
and the reporting of plans and results. 
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How Canada Health Infoway is funded 


4.20 The federal government has granted a total of $1.6 billion to 
Infoway in a series of funding agreements (Exhibit 4.2). 


Exhibit 4.2 Funds provided to Infoway by the federal government 


Amount 

Year - ($ millions) Purpose 

b : \ : a : - 

2001 500 Provide support for electronic health records (EHRs) 

and Telehealth 

2003 600 Provide additional support for EHRs and Telehealth 
2004 100 Provide support for public health surveillance 

2007 400 Continue work on EHRs and related priorities (such as 


wait times support) 


Source: The Canada Health Care, Early Childhood Development and Other Social Services nae 
Act, 2000, and Budget Implementation Act, 2003, 2004, and 2007. 


4.21 As Exhibit 4.1 shows, Infoway had spent about $614.9 million 
and committed just over $614.2 million in funding in support of EHRs 
as of March 2009. Infoway has also expended $212.4 million and 
committed another $134.7 million for other programs such as Patient 
Access to Quality Care, Public Health Surveillance, and Telehealth. 
The Budget Implementation Act, 2009, authorizes the payment of 

$500 million to Infoway for the implementation of EHRs. Some of the 
funds would go toward implementing EMR systems for doctors and 
connecting patients and points of service such as hospitals, community 
care facilities, and pharmacies. However, there is no funding 
agreement yet for this amount. 


Focus of the audit 


4.22 Our audit focused on whether Infoway is exercising due regard in 
managing funds from the federal government to achieve its goal 
related to implementation of EHRs that will be compatible across 
Canada, and whether Health Canada is ensuring that Infoway is 
complying with the funding agreements. We examined funding of 
projects in the five core systems that make up an EHR: registries, 
diagnostic imaging systems, drug information systems, laboratory 
information systems, and interoperable electronic health records 


(iEHRs). 


4.23 We also examined the governance structure, selected 
management controls (executive compensation, executive travel and 
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hospitality, and contracting for goods and services), and the 
appropriateness of Health Canada’s accountability arrangements under 
the various funding agreements. For the most part, our audit covered 
the 2006-07 and 2007-08 fiscal years. In some instances, such as the 
review of funded projects, we went back further. 


4.24 We did not audit other systems, such as Infrastructure, 
Innovation and Adoption; Patient Access to Quality Care; Public 
Health Surveillance; and Telehealth because they are not among the 
five core systems of an EHR listed in Exhibit 4.1. 


4.25 More details on the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


Setting the direction 4.26 Infoway’s funding agreements call for the Corporation to provide 
effective management and stewardship of the $1.6 billion allocated to 
it since 2001. We examined the governance mechanisms that allow 
Infoway to carry out its mandate and objectives. We expected Infoway 
to have clearly defined roles and responsibilities for the parties within 
the governance structure, and to have an independent Board of 
Directors that uses appropriate practices in exercising its stewardship 
role. 


Appropriate governance mechanisms are in place 


4.21 We reviewed documentation related to the governance 
structure, including the Board of Directors’ Charter, the terms of 
reference for the Board’s committees, and the Board and Committee 
minutes and packages since March 2006. As well, we interviewed the 
Chair and the Vice-Chair of the Board, and two other directors. We 
found that the roles and responsibilities are clearly defined 


(Exhibit 4.3). 


4.28 The Board’s role is to provide independent judgment on a broad 
range of issues. It is responsible for, among other things, ensuring that 
the Corporation operates in a manner that preserves its financial 
integrity and in accordance with the by-laws, funding agreements, 
applicable laws, and policies approved by the Board. Board members 
are required to comply with all corporate policies, including the Code 
of Business Conduct and the Conflict of Interest Policy. In line with 
this, directors sign forms each year in which they attest to their 
independence. Directors we interviewed informed us that those who 
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Exhibit 4.3 Roles of parties in Infoway’s governance 


Members 


Federal, provincial, and territorial deputy 
ministers of health 


¢ Appoint external auditors and review financial statements 


¢ Appoint seven directors to the-Board, who may but need not be government 
employees 


* Elect six directors at large, who shall not be government employees 


Board of Directors 
Two federal appointees 
Five provincial or territorial appointees 


Six elected directors at large 


¢ Reviews and approves, as need be, the organizational structure, strategic planning 
processes, risk management, and corporate policies 


Oversees finance and audits 


¢ Provides advice and counsel to management on critical or sensitive issues 


Appoints the President and Chief Executive Officer (CEO) and approves his or her 
compensation 


Approves funding of projects over a monetary threshold and oversees the quality 
and integrity of projects funded 


Board Committees 


Governance Committee 


Finance, Investment, and Audit 
Committee 


Compensation and Human Resources 
Committee 


e Assists the Board in fulfilling its obligations by providing a focus on corporate 
governance, with a view to enhancing the Board’s performance 


e Assists the Board in fulfilling its oversight of the financial affairs of the Corporation, 
investment of the financial assets, effective internal controls and financial reporting, 
audits and risk management functions, public accountability, and corporate 
compliance with laws and regulations and the Code of Business Conduct 


° Serves as an advisory committee and assists the Board in discharging its 
responsibilities that relate to compensation, organization structure, and executive 
staffing 


Officers and Management 


President and CEO and executive team 


e Executes strategies and plans approved by the Board 


e Is accountable for meeting performance targets and goals set by the Board 


Source: Adapted from Canada Health Infoway, Annual Report 2007-2008, page 29 


declare a potential conflict of interest do not participate in any 
discussion of that issue at Board meetings, nor do they vote on the 
matter. Our review of the meeting minutes supports this assertion. 
Thus, we found that the Board of Directors demonstrates 
independence and that appropriate measures are in place to deal with 
conflicts of interest. 


4.29 
practices in exercising its stewardship role, we conducted a literature 
search and identified good practices for boards. We noted that Infoway’s 
Board and committees follow several of these practices, such as 


In order to determine whether the Board uses appropriate 


* reviewing and approving strategic plans; 
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* monitoring Infoway’s performance; 


* overseeing management performance evaluations and 
compensation; 


¢ obtaining independent advice on compensation packages; 


¢ holding in camera sessions, without management present, when 
issues such as executive compensation and audits are discussed; 


* reviewing the succession plan for the executive team; 
* ensuring that appointments to the Board are staggered; and 


* periodically reviewing the performance of the Board. 


4.30 Based on our audit work, we concluded that Infoway’s Board of 
Directors uses appropriate practices in exercising its stewardship role. 


Infoway’s strategic plans and activities are consistent with the funding agreements 


4.31 The funding agreements require Infoway to prepare an annual 
corporate plan including a budget, objectives, and expected outcomes. 
In order for Infoway to achieve its objectives, it must develop and 
implement strategic plans and activities that will lead to electronic 
health records (EHRs) that are compatible across the country and 
consistent with the expectations of the federal, provincial, and 
territorial governments. | 


4.32 We expected Infoway to implement strategic plans based on the 
funding agreements. We also expected Infoway to implement 
operational plans to realize its strategic plans. 


4.33. We examined Infoway’s strategic documents, such as the 
corporate business plans. (for the 2007-2008, 2008-2009, and 
2009-2010 fiscal years), the Blueprint, and Infoway’s strategic 
framework, Vision 2015. We found that Infoway produces strategic 
plans consistent with the purpose, objectives, and conditions set out in 


the funding agreements. These plans identify priorities along with 


measurable goals and targets. The Corporation has also set indicators 
and targets to measure progress on achieving the goals of these plans. 


4.34 We also found that Infoway has developed an approach to 
implementing EHRs, using the funding agreements with Health 
Canada as a starting point. Infoway has annual operational plans and 
budgets, and investment strategies for each EHR system. These 
documents are consistent with the strategic plans and have measurable 
goals and targets to assess performance. 
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Infoway’s goal for implementation of electronic health records needs to be clearly 
defined 


4.35 Infoway’s goal consists of two elements: 


¢ By 2010, every province and territory and the populations they 
serve will benefit from new health information systems that will 
help transform their health care system. 


¢ By 2010, 50 percent of Canadians and, by 2016, 100 percent will 
have their electronic health record available to their authorized 
héalth care professionals. 


4.36 Achievement of the first element means that every jurisdiction 
will have at least one system in place. With respect to the second 
element, Infoway officials told us that having EHRs “available” does 
not necessarily mean that they are being used or that they are 
compatible across the country. We did not find this clearly explained in 
Infoway’s business plans or annual reports. 


4.37 We are concerned that the way the 2010 goal is stated may lead 
Canadians to believe that, when achieved, 50 percent of them will 
have an EHR that is used by their health care professionals and shared 
within and, if necessary, across jurisdictions. 


4.38 Recommendation. To ensure Parliament and Canadians 
properly understand Infoway’s goal, the Corporation should further 
explain in its public reports what is meant by having an electronic 
health record available to authorized health care professionals. 


Infoway’s response. We agree with the recommendation and we will 
make additional efforts to elaborate on the goal of having an electronic 
health record available to authorized health care providers in our 
public reporting commencing with our 2009-10 Annual Report and/or 
the 2010-11 Corporate Business Plan. Implementation of electronic 
health records is a complex task, as is reporting on progress. We have 
made continuous improvements to progress reporting and will 
continue to enhance our reporting in the future. Infoway reports on a 
regular basis jurisdictional progress to making electronic health records 
available to Canadians. For example, we provide members of 
Parliament and Senators with copies of our Annual Report and 
Corporate Business Plan. We also post these materials on our website 
to help ensure the broadest possible distribution to Canadians. 
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Infoway identified risks to realizing its goal and strategic plans 


4.39 Developing and implementing EHRs that are compatible across 
Canada is a complex and challenging task that involves the 
collaboration of many stakeholders and the integration of thousands of 
existing health information systems. Identifying and effectively 
managing the risks to this initiative are critical if Infoway is to fulfill its 
mandate and achieve its goal. 


4.40 In accordance with the funding agreements, we expected 
Infoway to have identified corporate risks that need to be managed so 
that it can realize both its goal and the objectives set out in its strategic 
plans. 


4.41 We found that Infoway has developed a Corporate Risk 
Management Framework and a risk management system to address 
risks identified in its strategic plans. The Framework describes risks to 
the achievement of the Corporation’s goal, the likelihood of each risk 
occurring, and the potential impact of each risk. According to Infoway, 
its highest risks are 


¢ failure to achieve the 2010 goal; 
* privacy or security breaches of EHR systems; 


¢ failure to achieve interoperable, replicable, and reusable systems 
that implement standards; 


* vendor resistance to adoption of standards; 
¢ slow adoption of EHR systems by health care professionals; and 


* failure to deliver projects on time, within budget, and of the 
quality or performance expected. 


4.42 The Framework presents the mitigation strategies that Infoway 
can use, given its mandate, to address these risks. It is regularly 
updated and presented to and discussed by the Finance, Investment, 
and Audit Committee and the Board of Directors. 


Infoway has policies that govern spending but some controls need strengthening 


4.43 We examined Infoway’s policies with respect to executive 
compensation, executive travel and hospitality, and contracting for 
goods and services. Treasury Board policies do not apply to Infoway 
because it is not a government department or agency. Infoway has 
therefore established its own policies in these areas. We also examined 
the appropriateness of the management controls in place to ensure 
that these costs are managed in accordance with these policies. For 
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the 2007-08 fiscal year, compensation for the executive team was 
$3.9 million. Travel and hospitality for the four executives included in 
our sample was $266,000. The value of the contracts in 2007-08 for 
operating, program, and project costs, including the related 
amendments and extensions, was $13.2 million. 


4.44 Executive compensation. Infoway’s Compensation Policy states 
that it will develop and maintain salary practices and procedures that 
provide an effective means of controlling salary expenditures while 
being responsive to changing market conditions. Thus, we expected 
Infoway to have established executive compensation packages, in 
accordance with its policies, that reflect due consideration for 
comparable benchmarks and the principles of prudence and probity. 


4.45 We found that Infoway’s policies reflect the principles of 
prudence and probity and give due consideration to comparable 
benchmarks. We also found that the policies had been followed in 
establishing the compensation packages. Some of the supporting 
management controls over executive compensation include the 
following: 


* compensation is governed by a strategy and policy approved by 


the Board; 


¢ market surveys (to compare compensation rates in similar 
organizations) are conducted every two years by an independent 
compensation consulting firm; 


¢ the President and CEO’s compensation package is approved by 
the Board; and 


¢ individuals’ objectives are linked to corporate objectives at the 
start of each year and performance is assessed annually based on 
those objectives. 


4.46 We tested the management controls and the way the 
compensation policy was applied by examining the files of half of the 
executive team. We found that the controls were adequate to ensure 
compliance with the policies. 


447 We also examined the disclosure of information related to 
executive compensation. Section 7.1 of the 2007 Canada Health 
Infoway Funding Agreement with the federal government requires 
Infoway to disclose in its annual report the total amount paid to its 
management team. In its 2007-2008 Annual Report, Infoway stated 
that the total remuneration of the management team for the fiscal year 


ended 31 March 2008 was $3.9 million. This represents the base salary, 
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car allowance, performance bonus, retention bonus, and RRSP or 
pension contributions for nine full-time equivalents. 


4.48 Executive travel and hospitality. Infoway has established 
policies for travel as well as hospitality and gifts. Thus, we expected 
Infoway to ensure that travel and hospitality expenses are managed in 
accordance with its policies. We compared Infoway’s policies with 
those of the Treasury Board of Canada. While not binding on Infoway, 
the Treasury Board policies provide a relevant point of comparison. 
We noted that the travel policy provisions were similar in all important 
respects, except that Infoway’s policy requires verbal rather than 
written approval of the supervisor prior to travel. In comparing the 
hospitality policies, we noted that Infoway’s policy does not specify 
dollar limits as does the federal policy. Instead, Infoway’s policy states 
that the reasonableness of expenses should be in accordance with its 
Code of Business Conduct. 


449 We examined travel and hospitality claims selected at random 
from the 2007-08 fiscal year and found that the policies had been 
followed in all important respects and that appropriate management 
controls were in place. 


450 Contracting for goods and services. Infoway has established a 
procurement policy for the acquisition of goods and services. Thus, we 
expected Infoway to ensure that contracts for goods and services are 
managed in accordance with its policy. We were told that the 
procurement policy was adapted from several organizations’ policies, 
including the federal government’s. We compared Infoway’s policy 
with those of the Treasury Board of Canada. We noted that the federal 
policy specifies monetary thresholds above which contract 
amendments must be competitively awarded. Infoway, however, does 
not require that contract amendments be competitively awarded if the 
initial contract was competitively awarded. 


4.51 Infoway’s procurement policy and guidelines specify a number of 
management controls, including signing authorities and thresholds for 
approvals. They also require that management decisions to forgo 
competitive bidding be disclosed to the Finance, Investment, and 
Audit Committee of the Board of Directors. 


4.52 To test the effectiveness of these controls, we examined a sample 
of 30 contracts chosen at random and all 5 contracts that were exempt 
from competitive bidding, as per Infoway’s policy, because of the 
contractors’ unique and specialized expertise. These contracts were for 
goods and services obtained in the 2006-07 and 2007-08 fiscal years. 


Report of the Auditor General of Canada—Fall 2009 


Chapter 4 17 


ELECTRONIC HEALTH RECORDS 


We found control weaknesses that create an increased risk of contract 
disputes or deliverables that may not meet Infoway’s expectations. For 
example, 


¢ 13 of the 35 contracts we reviewed were put in place after work 
began or was completed (often by several weeks); and 


¢ 14 of the 35 contracts in our sample were amended; of the 
14 contracts, 10 were amended three to seven weeks after the 
original contract or amendment had expired. 


4.53 Recommendation. Infoway should review and strengthen its 
management controls over contracting for goods and services to 
reduce the risk of contract disputes. 


Infoway’s response. We agree to review and analyze current practices 
to strengthen management controls. Improvements to management 
controls related to the timing of contract signatures will be put in place 
by the end of the 2009-10 fiscal year. We do not pay any invoice unless 
the contract is acceptably executed and the Infoway manager has 
certified that the product or service has been appropriately delivered 
by the contractor. 


4.54 Amending contracts for goods and services. Infoway’s policy 
does not require information concerning amendments to contracts 
awarded competitively to be reported to the Board’s Finance, 
Investment, and Audit Committee. We found that 8 of the 

35 contracts we examined were, in accordance with the policy, 
amended several times without using a competitive process. This 
raised their value significantly. In one particularly significant instance, 
we found that a one-year contract for $144,000, originally 


_ competitively awarded, was amended five times over two and a half 


years, raising its total value to $726,000. In our opinion, this practice is 
not conducive to the fair and transparent awarding of contracts and it 
raises questions about the appropriateness of Infoway’s contracting 
policy. Also, not reporting significant contract amendments to the 
Committee restricts its ability to monitor the effectiveness of internal 
controls and its members from exercising their duties as directors. 


455 Recommendation. Infoway should review its contracting policy 
with respect to contract amendments and extensions to ensure 
fairness, transparency, and disclosure to the Board. 


Infoway’s response. We agree to review the contracting policy with 
respect to amendments and extensions and to present the proposed 
revisions to the Board for approval prior to the fourth quarter of 


the 2009-10 fiscal year. 
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Infoway’s Board-approved contracting policy was developed to respond 
to competitive market conditions and the often unique and rare 
combination of skill sets required to deliver the Corporation’s 
mandate. The current policy reflects the necessary balance between 
market competiveness, the Corporation's business requirements, and 
due respect for the expenditure of public funds. We are compliant with 
the current policy related to the tendering of amendments and 
extensions and the reporting of those amendments and extensions to 
the Board of Directors. 


4.56 Since it was founded in 2001, Infoway has approved about 

$1.2 billion in funding for projects related to the five core systems of an 
electronic health record (EHR) (Exhibit 4.1). Most of these projects 
are cost-shared with the provinces and territories and involve projects 
critical to the success of Infoway’s goal of 50 percent of Canadians 
having an EHR available to their authorized health care professionals 


by 2010 and 100 percent by 2016. 


4.57 Infoway approves each project based on the project’s potential 
for success and commitment to satisfying requirements, such as 
aligning with the Blueprint and complying with standards. It funds 
projects using a gated funding model that ties reimbursement to the 
achievement of project milestones. As mentioned in paragraph 4.12, 
for most EHR projects, 50 percent of the funding Infoway reimburses 
to the provinces and territories is directly tied to adoption of systems 


that it funded. 


4.58 We reviewed 29 projects approved by Infoway. We chose the 
sample from those projects that are deemed to be core systems for an 
EHR: registries, diagnostic imaging systems, drug information systems, 
laboratory information systems, and interoperable electronic health 
records (EHRs). From the Corporation’s inception in 2001 to 

31 March 2009, the projects we chose to review accounted for 

$886 million or 72 percent of funding approved by Infoway for 
investment in the five core systems. 


Infoway approves projects that are consistent with its strategic plans 


4.59 We expected Infoway to ensure that it funds projects consistent 
with its strategic plans and, most importantly, the Blueprint. 


4.60 We found that Infoway has established and consistently applied 
an approval process that assessed proposed projects against the 
requirements for EHRs that are compatible across the country. The 
Corporation has also established legal agreements for each project with 
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the provinces and territories, to provide Infoway with greater 
assurance that the requirements will be respected. 


4.61 ‘lo ensure that projects address issues related to privacy and 
security, and to satisfy the terms of its funding agreements with Health 
Canada, Infoway requires jurisdictions to submit, or make available for 
review, privacy impact assessments (PIAs) for projects that will 
ultimately involve personal health information. Infoway examines 
these assessments to ensure that projects have addressed their privacy 
and security requirements. In those projects we examined that 
required PIAs, we found that the jurisdictions had submitted them, or 
made them available, to Infoway for review. 


4.62 We also found that project documentation submitted to Infoway 
at the time of approval outlined how all the projects we reviewed 
would comply with standards and align with the Blueprint, or would do 
so over time. Requirements related to standards and the Blueprint 
were also included in project statements of work. 


Infoway monitors the progress of projects but needs to better document analyses of 
project deliverables 


4.63 We expected Infoway to monitor the progress of EHR projects to 
ensure that project objectives are achieved and project risks are 
managed. 


4.64 We found that Infoway has established a process for monitoring 
its funding of projects, but the process could be better documented and 
more steps could be taken to ensure that its requirements are satisfied. 


465 Documentation of project deliverables and analyses. Infoway 
obtains most of its information on the progress of its projects through 
progress reports submitted by the provinces and territories. It also stays 
informed of progress by reviewing project deliverables, being part of 
project steering committees, and communicating informally with the 
provinces and territories. 


4.66 Infoway’s project managers compile the information obtained 
through these methods into monthly project status reports. These 
reports assess how well a project is progressing against the approved 
plan. They also provide a current assessment of various risks and issues 
that may have an impact on the project. 


467 We reviewed project status reports prepared by Infoway and 
found that project management teams are monitoring issues and risks. 
We also found that Infoway had established and consistently followed 
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its process for reviewing and approving project deliverables. However, 
we noted that analyses to support the approval of project deliverables 
were not always fully documented. Because the approval of project 
deliverables is tied to the release of funds, often worth hundreds of 
thousands of dollars, analyses to ensure that the project requirements 
have been met and that acceptable quality has been achieved ought to 
be better documented to ensure accountability and transparency. 


468 Recommendation. To ensure accountability and transparency, 
Infoway should better document its analyses of project deliverables to 
support its decision to release funds. 


Infoway’s response. We agree that there is scope for improvement to 
deliverables documentation and the Corporation will address this issue 
immediately. This will build on Infoway’s documentation process and 
methodology for the approval of deliverables, including the third-party 
claims verification process that assesses the adequacy and effectiveness 
of the policies, controls, and systems in place for the management of 
Infoway-funded projects. 


Infoway does not have sufficient assurance that standards are correctly 
implemented 


469 Conformance testing. As a condition of funding, Infoway 
requires projects to comply with standards and align with the 
Blueprint. According to Infoway, projects must comply with these 
requirements in order to achieve EHRs that are compatible across the 
country. 


4.70 To ensure that projects have satisfied these requirements, it is 
important that systems be tested. Infoway has acknowledged that it 
must verify compliance with standards to achieve its vision of EHRs 
that are compatible across the country. It has informed its Board of 
Directors that the failure to correctly implement standards poses risks 
to high-quality health care, patient safety, and the privacy of patient 
information. Infoway told us that it has a plan and an approach to 
ensure the compatibility of systems across Canada in those instances 
where there are variances in standards conformance. However, we 
noted that the approach to dealing with these variances has not yet 
been fully developed. Consequently, we believe the inconsistent 
implementation of standards remains a risk to national compatibility. 
Infoway’s own assessment of risk regarding the compatibility of EHR 
systems is rated as high. 


4.71 Conformance testing determines if a standard or specification 
has been correctly implemented in a system. Because standards may be 
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interpreted and implemented in different ways, conformance testing 
can help reduce the risks associated with the improper implementation 
of standards. 


4.72 Infoway acknowledges the value of conformance testing and has 
taken a number of steps to support testing by the provinces and 
territories. For example, the Corporation has defined guidelines for the 
implementation of standards. It has also developed tools that the 
provinces and territories can use to test their systems. 


4.73 However, we found that Infoway does not obtain from the 
provinces and territories the results of conformance testing on EHR 
systems. Consequently, the Corporation does not have sufficient 
assurance that EHR systems will be correctly implemented. 


4.74 Recommendation. To ensure that standards will be properly 
implemented in the five core systems of an electronic health record, 
Infoway should obtain from the provinces and territories the results of 
conformance testing on systems it will fund, and obtain assurance that 
non-conformance issues, if any, will be resolved. 


Infoway’s response. On a go forward basis, for new investment 
approvals, and with the cooperation of the jurisdictions, we agree to 
work to obtain the results of conformance testing from the jurisdictions 
on the five core electronic health record systems we will fund in the 
future and obtain assurance that non-conformance issues, if any, will 
be resolved. 


We have acknowledged that there will be variations in systems 
requirements and thus standards requirements within jurisdictions. 
Jurisdictions have the mandate and authority to ensure conformance 
and perform user acceptance testing to their specific requirements. 
They are most familiar with those requirements and are best positioned 
to ensure they are met. Some differences are valid within a 
specification and require no action, whereas some are due to different 
maintenance releases of a standard used by various jurisdictions largely 
driven by their timeline for implementation. Despite variances in 
standards conformance, the use of a standard within a jurisdiction will 
ensure interoperability within that jurisdiction. 


Pan-Canadian interoperability can still be achieved and differences in 
standards can be mitigated by 


¢ ensuring the same standard is used for cross-jurisdictional 
interoperability, or . 
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* mapping to accommodate differences in the implementation of 
the standard. 


We believe that this is not only feasible but also practical and cost 
effective. 


Infoway has taken reasonable steps to maximize its use of funds for projects 


4.75 We expected Infoway, in accordance with the funding 
agreements with Health Canada, to show that it collaborates with its 
federal, provincial, and territorial partners and other stakeholders to 
ensure the best use of funds for EHR projects. 


476 We found that Infoway has taken reasonable steps to maximize 
the use of funding. The Corporation has developed and supports 
initiatives that make it possible for its partners to replicate and reuse 
information and knowledge in their EHR projects. As mentioned 
before, foremost among these are the Blueprint and the Standards 
Collaborative. 


4.11 Infoway also promotes the reuse of common systems through 
agreements with preferred providers. This is designed to reduce costs, 
encourage replication, streamline the Corporation’s project approval 
process, and ensure that projects adhere to requirements for national 
compatibility. Agreements with preferred providers are established 
only after the vendor has successfully completed a provincial or 
territorial procurement process or a procurement process coordinated 
by Infoway on behalf of the provinces or territories. 


4.78 Infoway has developed sets of web-based toolkits that contain a 
variety of project documents such as case studies, templates, plans, and 
reports. The toolkits are designed to help reduce the risk, time, and 
cost of implementing projects. 


479 As mentioned earlier, Infoway has developed a measurable goal 

and targets to assess its progress in developing and deploying electronic 
health records (EHRs) across the country. These provide the basis for 
reporting on progress it has achieved. 


480 The funding agreements acknowledge that Infoway’s ability to 
achieve key outcomes depends on collaboration with the provinces 
and territories. Although Infoway needs provinces and territories 
representing 50 percent of Canada’s population to implement each 
EHR system to achieve its 2010 goal, it relies on the provinces and 
territories to set their own pace for implementation. 
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481 In accordance with the 2007 Funding Agreement’s requirement 
regarding ongoing performance monitoring strategies, we expected 
Infoway to have selected performance indicators and targets to 
measure progress on achieving the goal identified in its strategic plans 
and to report on progress achieved compared with expected 
performance. 


4.82 We reviewed Infoway’s corporate goal and targets from 
2005-2006 to 2009-2010. We examined its public reporting and 
internal reporting against this goal and these targets. We also analyzed 
the methods it has used to describe and report progress. 


Additional information on progress achieved is needed 


4.83 Reporting on progress achieved. Reporting progress on a 
complex task such as creating EHRs that are compatible across the 
country is difficult. Infoway reports progress against its 2010 goal by 
measuring the percentage of Canadians living in provinces or 
territories where an EHR is available to their health care professionals. 
As each province or territory implements all the core systems required 
for an EHR, its percentage of the total Canadian population is counted 
toward Infoway’s 2010 goal. In its 2009-2010 Corporate Business Plan, 
Infoway reported that 17 percent of Canadians live in provinces or 
territories where an EHR is available to their authorized health care 
professionals. More than half of these Canadians live in Alberta. 


4.84 Infoway’s 2010 goal concerns the availability of EHRs. Thus, it 
reports as complete those projects that are available for use. However, 
the Corporation has not publicly reported progress on other areas 
critical to the achievement of EHR systems across the country—for 
example, the adoption or use of systems by health care professionals. 


4.85 We found that although Infoway allocates 50 percent of its 
project funding to the achievement of adoption targets outlined in 
legal agreements for projects, the Corporation does not report on the 
adoption or use of systems it funds in the 2008-2009 Annual Report 
or 2009-2010 Corporate Business Plan. We found that some projects 
reported as 95 to 100 percent complete with respect to availability had 
not fully achieved adoption targets or were not being used. For 
example, the Alberta drug information system is available to health 
care professionals, but Infoway’s internal reporting indicates that the 
project has been on hold for about one year because the system is not 
using the pan-Canadian standard required to fully achieve the project’s 
adoption targets. Quebec’s client registry is also available for use, but 
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Infoway’s project status reports indicate that the project is facing a 
significant risk related to the implementation and use of the system. 


4.86 We believe that by reporting on the adoption or use of the EHR 
systems it funds, Infoway would provide assurance to Parliament and 
Canadians that the new systems are not only available but are 
beginning to be used as envisioned. 


487 We found that Infoway has not publicly reported on another 
critical area—the extent to which systems comply with standards and 
align with the Blueprint. A core principle of Infoway’s funding 
agreements with Health Canada is to work to achieve EHRs that are 
compatible across the country. Although not required under the 
funding agreements, Infoway ought to report on the extent to which 
completed systems meet the requirements for national compatibility in 
order to provide a better sense of progress. This information would 
provide additional insight into whether the objectives of the funding 
agreements are being achieved. For example, Infoway is reporting in 
its 2009-2010 Corporate Business Plan that a completed EHR system 
is available to all health care professionals in Prince Edward Island and 
Alberta. Although both provinces have made EHRs available to health 
care professionals, not all core systems of their EHR have implemented 
the standards necessary for national compatibility. In addition, both 
provinces will need to make additional investments before they are 
able to exchange information with other provinces and territories. 
Obtaining the results of conformance tests would help Infoway to 
report on compliance with standards. 


4.88 Recommendation. To ensure Parliament and Canadians have 
sufficient information about progress achieved, Infoway should report 
on the extent to which electronic health record systems have been 
adopted by health care professionals and are compliant with standards. 


Infoway’s response. We agree with the recommendation, recognizing, 
however, that solution implementation and availability must be 
completed before adoption can occur. These major project initiatives 
often take 24 to 36 months to be fully implemented. . 


Infoway, in conjunction with the jurisdictions, will endeavour to 
develop measures on the extent to which EHR systems have been 
adopted by health care providers. Further, Infoway will work with 
jurisdictions to develop a standards compliance report. 


489 Reporting results achieved compared with expected results. 
Infoway reports annually on expected results for the coming year and 
on the performance results of the previous year. However, we found 
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that the Corporation does not compare results achieved with expected 
results in either its corporate business plans or annual reports. Nor 
does it explain differences between its achievements and its 
expectations. For example, in the 2008-2009 Corporate Business Plan, 
Infoway expected that 28 percent of Canadians would have an EHR 
available to their health care-provider by 31 March 2009. According to 
the 2008-2009 Annual Report and the 2009-2010 Corporate Business 
Plan, the progress Infoway reported was 17 percent, or 11 percentage 
points less than expected. Infoway provides no indication in its 
Corporate Business Plan or Annual Report that it has fallen short of 
expected progress, and it does not explain the reasons for the shortfall. 


4.90 Recommendation. To ensure Parliament and Canadians have 
sufficient information about progress achieved, Infoway should report 
on results achieved compared with expected results and explain any 
difference. 


Infoway’s response. We agree to report on variances between 
expected results and achievements related to the electronic health 
record goal line in our public reporting beginning with the 2009-10 
Annual Report. 


This will build on the Corporation’s existing reporting on expected 
results and actual performance in its Annual Report (in the section 
entitled, Delivering Results: Performance Against Objectives). We 
provide members of Parliament (MPs) and Senators with copies of our 
Annual Report and Corporate Business Plan. Last year, we undertook 
an initiative (which we plan to repeat annually) to provide MPs and 
Senators with jurisdictional fact sheets to highlight EHR 
advancements within their respective region. 


491 Reporting on progress in achieving performance targets. 
Infoway has developed performance targets for each core system of an 
EHR. For example, by 2010, Infoway is expecting laboratory 
information systems to capture and store 75 percent of information on 
laboratory test results. By 2016, it expects to provide 95 percent of 
hospital and public health laboratories in eight provinces and 
territories with access to patients’ laboratory test profiles. Infoway is 
also expecting drug information systems to electronically capture and 
store 75 percent of information on dispensed medication. 


4.92 We found that although Infoway has reported these targets 
publicly in its 2007-2008 and 2008-2009 corporate business plans, the 
Corporation has not reported its progress in achieving the targets. 
Reporting on the extent to which health care professionals are using 
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EHR systems would provide useful information to Infoway 
management and more complete information to Parliament and 
Canadians. 


4.93 Recommendation. To ensure Parliament and Canadians have 
sufficient information about progress achieved, Infoway should report 
on the results achieved for performance targets established for each 
core system of the electronic health record. 


Infoway’s response. We agree with the recommendation. The 
Corporation will need to finalize its work with the jurisdictions on both 
data availability and data quality, to ensure that additional reporting 
will be accurate. This will build on the Corporation’s existing reports 
on the value, benefits, and advancement that electronic health record 
systems are delivering to Canadians through our.proactive media 
relations efforts, corporate materials, and the Internet, as well as 
through hundreds of public presentations and speaking engagements. 


494 Health Canada’s funding of Infoway is provided through a series 
of grants (made in 2001, 2003, 2004, and 2007). Accountability 
measures that apply to Infoway are defined in the funding agreements 
between the Corporation and Health Canada. These measures are 
intended to ensure that Health Canada is able to account to 
Parliament for Infoway’s use of funding provided by the federal 
government. 


4.95 The terms and conditions of funding agreements between Health 
Canada and Infoway have evolved over time as the expectations of 
governments have changed. Funding that Infoway received 

in 2001, 2003, and 2004 totalling $1.2 billion was provided in a lump 
sum transfer of funds, and use of the funds was subject to fewer 
reporting and evaluation requirements. 


4.96 The 2007 Infoway funding agreement included several new 
provisions to strengthen Infoway’s accountability to Health Canada. 
For example, under the 2007 funding agreement, funding to Infoway of 
$400 million is to be provided only on a needs basis. The funding 
agreement requires Infoway to submit annual cash flow statements 
estimating its financial requirements for the upcoming year. As of 
31 March 2009, Infoway had drawn down about $162 million in 
funding against the 2007 agreement. The agreement also allows the 
Minister to conduct an evaluation of Infoway’s activities and to 
request additional information for the purpose of keeping the 
Accounts of Canada. | 
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Health Canada needs to further develop its approach to monitoring 


497 Since 2001, the federal government has granted to Infoway a 
total of $1.6 billion in a series of funding agreements. Health Canada’s 
role as the federal signatory on the funding agreements is to ensure 
that Infoway is complying with the funding agreements. We therefore 
expected that Health Canada would have appropriate mechanisms in 
place to ensure that Infoway is complying with the funding 
agreements. 


4.98 Health Canada periodically obtains assurance of compliance 
with the funding agreements. Per the funding agreements, Infoway 
hires an independent auditor to conduct an annual compliance audit 
of its compliance with the terms of the funding agreements. Also, 
independent evaluations are conducted every five years. We did not 
audit any of these independent audits and evaluations. These audits 
and evaluations are the most important sources of information for 
ensuring that Infoway is complying with the funding agreements and is 
achieving objectives outlined in the agreements. All compliance audits 
to date, and the evaluation completed in 2006, have concluded that 
Infoway is complying with the funding agreements in all important 
respects and is making progress toward its objectives. 


4.99 Health Canada also needs to regularly monitor Infoway’s 
activities so that it has early warning of potential non-compliance. 
Although Infoway was created in 2001, Health Canada has just 
approved, in 2008, a monitoring framework to help the Department 
and the federal government reduce the inherent risk associated with 
such large amounts of funding. The framework identifies Infoway’s 
accountability requirements, as outlined in the funding agreements. 
It also identifies 14 processes necessary to assess the Corporation’s 
compliance with those requirements. At the time of the audit, eight of 
these processes had been developed. For example, the monitoring 
framework describes the process to follow in reviewing annual reports 
and corporate business plans. 


4.100 We noted that at the time of the audit, Health Canada had not 
yet developed the remaining six processes. For example, the 
Department had identified the need for a risk assessment tool to 
identify and assess areas of risk to compliance with the funding 
agreements, but it had not yet developed one. 


4.101 Recommendation. To better fulfill its responsibility for 
monitoring Infoway’s compliance with the funding agreements, Health 
Canada should fully develop and implement its monitoring framework. 


Report of the Auditor General of Canada—Fall 2009 


Meeting important challenges 


ELECTRONIC HEALTH RECORDS 


Health Canada’s response. Health Canada agrees with the Auditor 
General’s recommendation. 


The Department recognizes the importance of monitoring Infoway’s 
compliance with the funding agreements and has been monitoring its 
compliance since the first funding ‘agreement in 2001. As of 
September 2009, the Department has developed a comprehensive 
framework to monitor Infoway’s compliance more systematically. The 
framework includes an overview of Infoway’s deliverables and 
activities for a given fiscal year, schedules with the timing of annual 
and ad hoc deliverables and activities, and a description of federal 
activities to support due diligence. The framework also includes 

14 appendices that set out detailed procedures and tools used to 
support monitoring, including a risk assessment tool and checklists to 


foster consistency and transparency (such as factors to consider in 


reviewing the Annual Report), and procedures to be followed for 
specific events and activities (such as ministerial appointments to the 
Board). The framework will be updated periodically, as needed. 


The Department welcomes the Auditor General’s support for our 
approach to monitoring investments in Infoway. 


4.102 Our chapter notes that even though Infoway has made a 
significant contribution to the development and implementation of 
EHRs, other important challenges remain. These include ensuring that 
systems have been adopted and that tangible benefits of EHRs are 
realized throughout the country. Understanding these challenges 
provides the context for understanding the major task facing Infoway, 
Health Canada, and the provinces and territories. Thus, this section is 
forward-looking and is not based on our audit work. It is intended to 
help readers of this report to put the observations noted in the chapter 
in perspective. 


Meeting the 2010 goal 


4.103 Infoway is facing major challenges in reaching its 2010 goal of 
50 percent of Canadians having an EHR available to their authorized 
health care professionals. As previously noted, Infoway is dependent 
on the provinces and territories to secure funds and implement the 
projects. According to Infoway, as of 31 March 2009, only 17 percent 
of Canadians lived in a province or territory where a complete EHR 
system was available. Infoway anticipates that it will achieve the 

50 percent goal by 31 December 2010. At the same time, we noted 
that Infoway’s own assessment of the risk to achieving the goal is high. 
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Upgrading systems to achieve compatibility 


4.104 Not all completed EHR projects have implemented the 
standards required for national compatibility. According to Infoway, 
some of these projects will do so only after they have been upgraded as 
part of future projects. It is not clear who will provide the funding 
needed to ensure that all systems will be compatible or when this 
funding will be provided. 


Implementing electronic medical records 


4.105 Infoway has estimated that about 80 percent of patient 
encounters with the health care system take place with doctors and 
specialists outside a hospital setting: A survey conducted in 2007 
found that approximately 10 percent of Canadian doctors maintain » 
their patients’ health records electronically. Another 26 percent of 
doctors use a mix of electronic and paper-based records. Unless the 
percentage of primary care doctors using electronic medical records 
(EMRs) increases significantly, the potential benefits offered by 
electronic health records (EHRs) will not be fully realized (see 
paragraph 4.4 on the difference between an EMR and EHR). As 
previously mentioned, the Budget Implementation Act, 2009, authorizes 
the payment of $500 million to Infoway. However, there is no funding 
agreement yet for this amount. Some of these funds would go toward 
the implementation of EMR systems. 


Using new consumer health solutions 


4.106 New health information systems designed to allow patients direct 
access and control over their personal health information are 
becoming more common. These systems are often referred to as 
consumer health platforms or solutions. It is expected that patients will 
use these platforms or solutions to share personal health information 
with health care professionals as well as with EHRs, EMRs, and 
hospital information systems. Companies have indicated their intent to 
provide consumer health solutions to Canadians. 


4.107 As mentioned previously, nationally compatible EHRs require 
that all components of the system be standards-based and comply with 
compatibility requirements. Consumer health solutions are no 
different. It will be important that the vendors of these systems offer 
solutions compatible with the systems Infoway has funded to date. 


4.108 Infoway has begun to address this challenge by launching a 
certification service to ensure that consumer health solutions available 
for sale meet the Corporation’s requirements for privacy, security, and 
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compatibility. Certification is expected to assure individual Canadians 
and other stakeholders, such as doctors and pharmacies, that 
consumer health solutions from certified vendors are compatible with 
existing EHR systems. 


4.109 According to Infoway, the success of its certification service will 
ultimately depend on the extent to which provinces and territories, 
hospitals, doctors, pharmacists, laboratories, and suppliers of software 
products comply with standards themselves. This dependency 
accentuates the importance of Infoway’s continuing to approve 
projects that commit to complying with standards and taking steps to 
verify that EHR projects implement those standards. 


Ensuring the privacy and security of personal health information 


4.110 The concept of EHRs that are compatible across the country is 
based on the premise that Canadians’ health information will be 
available to their health care professionals no matter where they live or 
where they may be when they need health care services. The sharing of 
data among the provinces and territories is critical to the achievement 
of EHRs that are compatible across the country. This is particularly 
true for Canadians who live in smaller provinces and territories and 
require specialized care available only in larger centres. It is also of 
concern to Canadians who move from one part of the country to 
another, travel often, or live in one place but work or study in another. 


4.111 Different provincial and territorial laws regarding the collection, 
use, and disclosure of personal health information pose a challenge to 
the sharing of this information between jurisdictions. Although 
Infoway is not responsible for resolving these legislative differences, the 
Corporation has set up discussions between provincial and territorial 
ministries of health, and agencies in each province and territory 
responsible for ensuring that provincial and territorial privacy laws are 
respected. 


4.112 Together with the Privacy Forum, Infoway has established 
working groups to focus efforts on common approaches for 
accountability, consent, secondary uses, and flow of data between 
jurisdictions. 


4.113 Finally, a privacy breach of EHR systems has the potential to 
undermine the strong support among Canadians for the EHR 
initiative. Such a breach would also negatively affect support for EHRs 
among health care professionals. 
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Conclusion 


4.114 Overall, we found that Canada Health Infoway is exercising due 
regard in managing funds from the federal government to achieve its 
goal related to the implementation of electronic health records (EHRs) 
across Canada. 


4.115 Our examination of EHRs shows that Infoway has set a good 
foundation for the work it is doing by applying appropriate governance 
mechanisms to carry out its mandate and objectives. It has also 
implemented strategic plans, such as the Blueprint, that are consistent 
with its funding agreements with Health Canada. These plans guide its 
investments in creating EHRs that are compatible across Canada. 
Infoway has set a goal for this initiative, but it could be more clearly 
defined. It has also implemented appropriate management controls for 
operational spending, although controls for contracting for goods and 
services need to be strengthened. 


4.116 We found that Infoway is exercising due regard in approving, 
monitoring, and making best use of its funds for EHR projects. 
However, Infoway ought to better document its analyses of project 
deliverables before releasing payments. Infoway approves projects that 
are designed to align with the Blueprint and comply with standards. 
However, it does not have sufficient assurance that, once 
implemented, projects will comply with these requirements. 


4.117 We observed that Infoway reports progress on achieving the goal 
identified in its strategic plans and the funding agreements. However, 
the Corporation does not annually report progress achieved against 
expected progress and it does not report on its performance targets 
that are related to the use of EHR systems. Also, it does not report on 
project adoption and project compliance with requirements for 
national compatibility. This information is important to providing a 
complete picture of progress achieved. 


4.118 Through annual independent compliance audits and periodic 
evaluations, Health Canada ensures that Infoway complies with the 
funding agreements. However, at the time of the audit, the 
Department had not yet fully developed an approach to its ongoing 
monitoring activities to detect early warning of potential 
non-compliance. 


4.119 Infoway has accomplished much in the eight years since its 
creation. Progress achieved thus far required the collaboration of 
Infoway, all provinces and territories, and other stakeholders. 
Significant challenges remain to ensuring that tangible benefits of 
EHRs are realized across the country. 
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About the Audit 


All of the audit work in this chapter was conducted in accordance with the standards for assurance 
engagements set by The Canadian Institute of Chartered Accountants. While the Office adopts these 
standards as the minimum requirement for our audits, we also draw upon the standards and practices of 
other disciplines. 


Objectives 


The overall audit objective was to determine whether Canada Health Infoway Inc. (Infoway) is exercising 
due regard in managing funds from the federal government to achieve its goal of implementing compatible 
electronic health records across Canada, and whether Health Canada ensures that Canada Health 
Infoway Inc. complies with the funding agreements. 


The audit sub-objectives were to determine whether 


- Infoway is implementing strategic plans consistent with the funding agreements to guide its funding of 
compatible electronic health records across Canada; 


- Infoway is exercising due regard in approving, monitoring, and maximizing its use of funds for 
electronic health records projects; 


- Infoway reports progress on achieving the goal identified in its strategic plans and the funding 
agreements; 


- Infoway has appropriate governance mechanisms to carry out its mandate and objectives; 
¢ Health Canada ensures that Infoway complies with the funding agreements; and 


- Infoway has appropriate management controls in selected areas. 


Scope and approach 


Our audit focused on Canada Health Infoway Inc.’s funding of compatible electronic health records 
(EHRs) across Canada. We reviewed 29 projects approved by Infoway. We chose the sample from those 
projects that are deemed to be the five core systems for an EHR: registries, diagnostic imaging systems, 
drug information systems, laboratory information systems, and interoperable electronic health records. 
Projects were selected in order to provide a high degree of coverage of total approved funding, as well as at 
least 50 percent of approved funding for each of the five core systems. As of 31 March 2009, the projects 
we chose to review accounted for $886 million or 72 percent of funding approved by Infoway for 
investment in the core systems. We did not audit other systems such as Infrastructure, Innovation and 
Adoption; Patient Access to Quality Care; Public Health Surveillance; and Telehealth. 


We examined the governance structure, selected management controls (executive compensation, 

executive travel and hospitality, and contracting for goods and services). We looked at 35 contracts out of 
a total of 555 contracts ranging in value from $179.89 to $6.8 million awarded between | April 2006 and 
31 March 2008. Of the 35 contracts, 30 were selected at random. The other 5 were selected because they 


had been exempted from competitive bidding, per Infoway’s policy. 
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Finally, we reviewed the appropriateness of Health Canada’s accountability arrangements under the 
various funding agreements. Due to the difficulty in obtaining reliable data, our audit did not attempt to 


compare Canada’s progress in developing and implementing electronic health records with that of other 


countries. 


Criteria 


Listed below are the criteria that were used to conduct this audit and their sources. 


Criteria 


Sources 


Canada Health Infoway Inc. 


We expected that Canada Health Infoway Inc. would have 
clearly defined roles and responsibilities of the parties in the 
governance structure. 


The Canadian Institute of Chartered Accountants, 
20 Questions Directors of Not-for-Profit Organizations Should 
Ask About Governance, page 19 


Excellence in Canada’s Health System; Principles for 
Governance, Management, Accountability and Shared 
Responsibility, page 14 


We expected that Canada Health Infoway Inc. would have an 
independent Board of Directors that uses appropriate practices 
in exercising its stewardship role. 


Health Canada Accountability Framework for Canada Health 
Infoway, page 23 


Canada Corporations Act, subsections 98(1) to (4) 


Canada Health Infoway, Board of Directors’ Charter #5 and 
#6 (21 September 2005) 


We expected that Canada Health Infoway Inc. would implement 
Strategic plans that are consistent with the funding agreements. 


Enterprise Value: Governance of IT Investments, Val IT 
Framework 2.0-VG1.5, page 33 


2007 Canada Health Infoway Funding Agreement, 
section 4.1, page 8 


2003 Canada Health Infoway Funding Agreement, 
section 3.4.1, page 6 


We expected that Canada Health Infoway Inc. would have 
operational plans to realize its strategic plans. 


Enterprise Value: Governance of IT Investments, Val IT 
Framework 2.0-VG2.4, page 35 


We expect Canada Health Infoway Inc. would have identified 
corporate risks that need to be managed to realize its strategic 
plans. 


2007 Canada Health Infoway Funding Agreement, 
section 1.5, page 3 


We expected that Canada Health Infoway Inc. would have 
established executive compensation packages, in accordance 
with its policies, that reflect due consideration for comparable 
benchmarks and the principles of prudence and probity. 


Canada Health Infoway, Board of Directors’ Charter 
(21 September 2005) 


Canada Health Infoway’s Compensation Policy 


We expected that Canada Health Infoway Inc. would ensure that 
travel and hospitality expenses are managed in accordance with 
its policies. 


Canada Health Infoway’s Travel Policy and Hospitality and 
Gifts Policy 


Canada Health Infoway, By-Law No.1 (18 January 2001) 


Canada Health Infoway, Signing Authorities 
(6 December 2006) 
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contracts for goods and services are managed in accordance 
with its policies. 
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Sources 


* Canada Health Infoway’s Procurement Policy 


e The Canadian Institute of Chartered Accountants, 
20 Questions Directors of Not-for-Profit Organizations Should 
Ask About Governance, page 16 


We expected that Canada Health Infoway Inc. would ensure that 
the projects it funds are consistent with its strategic plans and 
the EHRS Blueprint. 


e Enterprise Value: Governance of IT Investments, Val IT 
Framework 2.0-VG2.4, page 35 


We expected that Canada Health Infoway Inc. would monitor 
the progress of electronic health records projects to ensure that 
project objectives are achieved and project risks are managed. 


e Enterprise Value: Governance of IT Investments, Val IT 
Framework 2.0-VG5.3, IM6.2, and IM6.3, pages 41 and 81 


¢ 2007 Canada Health Infoway Funding Agreement, 
SCCUONS Sale SLO Oleh On2=S eanG.2.05 DacesalOlancd aint 


We expected that Canada Health Infoway Inc. would show that 
it collaborates with federal, provincial, and territorial partners, 
and other stakeholders, to maximize its use of funds for 
electronic health records projects. 


¢ 2007 Canada Health Infoway Funding Agreement, 
sections 5.1.3, 5.2.2, and 5.2.4, pages 10 and 11 


We expected that Canada Health Infoway Inc. would have 
identified performance indicators and targets to measure 
progress on achieving the goal identified in its strategic plans 
and the funding agreements. 


e Enterprise Value: Governance of IT Investments, Val IT 
Framework 2.0-VG5.1, page 41 


e 2007 Canada Health Infoway Funding Agreement, 
section 1.5 (v), page 3 


We expected that Canada Health Infoway Inc. would report 
progress achieved compared with expected performance. 


Enterprise Value: Governance of IT Investments, Val IT 
Framework 2.0— VG5.1 and VG5.3, page 41 


¢ 2007 Canada Health Infoway Funding Agreement, 
section 7.1.1(b), page 15 


e 2004 Canada Health Infoway Funding Agreement, 
sections 5.1.5 and 5.2.1.5, pages 13 and 14 


~ Health Canada 


We expected that Health Canada would have implemented 
appropriate mechanisms for ensuring that Canada Health 
Infoway Inc. complies with the funding agreements. 


¢ Health Canada Accountability Framework for Canada Health 
Infoway 


* 2007 Canada Health Infoway Funding Agreement, Preamble 


Management reviewed and accepted the suitability of the criteria used in the audit. 
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Period covered by the audit 


For the most part, our audit focused on the 2006-07 and 2007-08 fiscal years. In some instances, we went 


back further. 


Planning and reporting 
EHR projects 

Executive compensation 
Travel and hospitality 
Contracting 


Health Canada 


Audit work for this chapter was substantially completed on 30 April 2009. 


Audit team 


2005-06 to 2009-10 
2003-04 to 2009-10 
2006-07 to 2008-09 
2007-08 

2006-07 to 2007-08 
2001-02 to 2008-09 


Assistant Auditor General: Neil Maxwell 


Principal: Louise Dubé 
Director: Holly Shipton 


lan Campbell 

Mark Carroll 

Daphné Lamontagne 
Jean-Francois Luc 
Marie-Ange Massunken 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 4. The number in front of the 


recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 


indicate the paragraphs where the topic is discussed. 


Setting the direction 
4.38 To ensure Parliament and 
Canadians properly understand 
Infoway’s goal, the Corporation should 
further explain in its public reports 
what is meant by having an electronic 


health record available to authorized 
health care professionals. (4.35-4.37) 


4.53  Infoway should review and 
strengthen its management controls 
over contracting for goods and services 
to reduce the risk of contract disputes. 


(4.43-4.52) 


4.55 Infoway should review its 
contracting policy with respect to 
contract amendments and extensions to 
ensure fairness, transparency, and 


disclosure to the Board. (4.54) 


Infoway’s response. We agree with the recommendation and we 
will make additional efforts to elaborate on the goal of having an 
electronic health record available to authorized health care 
providers in our public reporting commencing with our 2009— 
10 Annual Report and/or the 2010-11 Corporate Business Plan. 
Implementation of electronic health records is a complex task, as 


is reporting on progress. We have made continuous 
improvements to progress reporting and will continue to 
enhance our reporting in the future. Infoway reports on a regular 
basis jurisdictional progress to making electronic health records 
available to Canadians. For example, we provide members of 
Parliament and Senators with copies of our Annual Report and 
Corporate Business Plan. We also post these materials on our 
website to help ensure the broadest possible distribution to 
Canadians. 


Infoway’s response. We agree to review and analyze current 
practices to strengthen management controls. Improvements to 
management controls related to the timing of contract 
signatures will be put in place by the end of the 2009-10 fiscal 
year. We do not pay any invoice unless the contract is acceptably 
executed and the Infoway manager has certified that the product 
or service has been appropriately delivered by the contractor. 


Infoway’s response. We agree to review the contracting policy 
with respect to amendments and extensions and to present the 
proposed revisions to the Board for approval prior to the 
fourth quarter of the 2009-10 fiscal year. 


Infoway’s Board-approved contracting policy was developed to 
respond to competitive market conditions and the often unique 
and rare combination of skill sets required to deliver the 
Corporation’s mandate. The current policy reflects the necessary 
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balance between market competiveness, the Corporation’s 
business requirements, and due respect for the expenditure of 


public funds. We are compliant with the current policy related to 


the tendering of amendments and extensions and the reporting 
of those amendments and extensions to the Board of Directors. 


Funding electronic health records projects 


468 ‘To ensure accountability and 
transparency, Infoway should better 
document its analyses of project 
deliverables to support its decision to 


release funds. (4.63—4.67) 


414 Toensure that standards will be 
properly implemented in the five core 
systems of an electronic health record, 
Infoway should obtain from the 

provinces and territories the results of 
conformance testing on systems it will 
fund, and obtain assurance that 

non-conformance issues, if any, will be 


resolved. (4.69-4.73) 
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Infoway’s response. We agree that there is scope for 
improvement to deliverables documentation and the 
Corporation will address this issue immediately. This will build 
on Infoway’s documentation process and methodology for the 
approval of deliverables, including the third-party claims 
verification process that assesses the adequacy and effectiveness 
of the policies, controls, and systems in place for the 
management of Infoway-funded projects. 


Infoway’s response. On a go forward basis, for new investment 
approvals, and with the cooperation of the jurisdictions, we 
agree to work to obtain the results of conformance testing from 
the jurisdictions on the five core electronic health record 
systems we will fund in the future and obtain assurance that — 
non-conformance issues, if any, will be resolved. 


We have acknowledged that there will be variations in systems 
requirements and thus standards requirements within 
jurisdictions. Jurisdictions have the mandate and authority to 
ensure conformance and perform user acceptance testing to 
their specific requirements. They are most familiar with those 
requirements and are best positioned to ensure they are met. 
Some differences are valid within a specification and require no 
action, whereas some are due to different maintenance releases | 
of a standard used by various jurisdictions largely driven by their 
timeline for implementation. Despite variances in standards 
conformance, the use of a standard within a jurisdiction will 
ensure interoperability within that jurisdiction. | 
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Pan-Canadian interoperability can still be achieved and 
differences in standards can be mitigated by 


* ensuring the same standard is used for cross-jurisdictional 
interoperability, or 


* mapping to accommodate differences in the implementation of 
the standard. 


We believe that this is not only feasible but also practical and 
cost effective. 


Reporting on results 


4.88 ‘To ensure Parliament and 
Canadians have sufficient information 
about progress achieved, Infoway 
should report on the extent to which 
electronic health record systems have 
been adopted by health care ~ 
professionals and are compliant with 


standards. (4.83-4.87) 


4.90 ‘To ensure Parliament and 
Canadians have sufficient information 
about progress achieved, Infoway 
should report on results achieved 
compared with expected results and 
explain any difference. (4.89) 


Infoway’s response. We agree with the recommendation, 
recognizing, however, that solution implementation and 
availability must be completed before adoption can occur. These 
major project initiatives often take 24 to 36 months to be fully 
implemented. 


Infoway, in conjunction with the jurisdictions, will endeavour to 
develop measures on the extent to which EHR systems have 
been adopted by health care providers. Further, Infoway will 
work with jurisdictions to develop a standards compliance 
report. 


Infoway’s response. We agree to report on variances between 
expected results and achievements related to the electronic 
health record goal line in our public reporting beginning with 


the 2009-10 Annual Report. 


This will build on the Corporation’s existing reporting on 
expected results and actual performance in its Annual Report 
(in the section.entitled, Delivering Results: Performance Against 
Objectives). We provide members of Parliament (MPs) and 
Senators with copies of our Annual Report and Corporate 
Business Plan. Last year, we undertook an initiative (which we 
plan to repeat annually) to provide MPs and Senators with 
jurisdictional fact sheets to highlight EHR advancements within 
their respective region. 
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4.93 To ensure Parliament and 
Canadians have sufficient information 
about progress achieved, Infoway 
should report on the results achieved 
for performance targets established for 
each core system of the electronic 


health record. (4.91-4.92) 


Being accountable to Parliament 


4.101 To better fulfill its responsibility 
for monitoring Infoway’s compliance 
with the funding agreements, Health 
Canada should fully develop and 
implement its monitoring framework. 


(4.974.100) 


Infoway’s response. We agree with the recommendation. The 
Corporation will need to finalize its work with the jurisdictions 
on both data availability and data quality, to ensure that 
additional reporting will be accurate. This will build on the 
Corporation’s existing reports on the value, benefits, and 
advancement that electronic health record systems are 
delivering to Canadians through our proactive media relations 
efforts, corporate materials, and the Internet, as well as through 
hundreds of public presentations and speaking engagements. 


Health Canada’s response. Health Canada agrees with the 
Auditor General’s recommendation. 


The Department recognizes the importance of monitoring 
Infoway’s compliance with the funding agreements and has been 
monitoring its compliance since the first funding agreement 

in 2001. As of September 2009, the Department has developed a 
comprehensive framework to monitor Infoway’s compliance 
more systematically. The framework includes an overview of 
Infoway’s deliverables and activities for a given fiscal year, 
schedules with the timing of annual and ad hoc deliverables and 
activities, and a description of federal activities to support due 
diligence. The framework also includes 14 appendices that set 
out detailed procedures and tools used to support monitoring, 
including a risk assessment tool and checklists to foster 
consistency and transparency (such as factors to consider in 
reviewing the Annual Report), and procedures to be followed for 
specific events and activities (such as ministerial appointments 
to the Board). The framework will be updated periodically, as 
needed. 


The Department welcomes the Auditor General’s support for 
our approach to monitoring investments in Infoway. 
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What we examined 


Why it’s important 


Acquiring Military Vehicles for Use 
in Afghanistan 


Main Points 


Canada’s military role in Afghanistan has been to contribute to 
international security and, in particular, to contribute to the stability of 
the area. National Defence has had to acquire new equipment urgently 
needed by the Canadian Forces to carry out their operations effectively 
and protect their personnel. 


We examined four urgent projects, each costing over $100 million, to 
acquire military vehicles that would improve operational capability 
and the protection of soldiers in Afghanistan. The projects involved 
replacement tanks, armoured patrol vehicles, armoured heavy support 
vehicles, and light armoured vehicles with remote weapon stations. 


We examined how National Defence managed the projects to ensure 
that the acquisitions met government policies related to project 
management and that the vehicles it was purchasing would meet the 
Canadian Forces’ urgent operational needs. We also examined how 
National Defence and Public Works and Government Services Canada 
(PWGSC) worked together to ensure that the contracting for the 


“projects complied with government policies. 


In addition, we looked at the analysis and challenge role played by the 
Treasury Board of Canada Secretariat when project and contract 
proposals were submitted for Treasury Board approval. 


Audit work for this chapter was substantially completed on 


29 May 2009. 


The Canadian Forces’ current deployment to Kandahar, Afghanistan has 
increased the risk to military and civilian personnel caused by threats 
such as improvised explosive devices. Failure to respond adequately and 
rapidly to these threats can have serious consequences—for example, 

it could leave the troops at greater risk or force Canada to adjust the 
scope of its military operations in Afghanistan. 


The ability to acquire new equipment rapidly is essential to help reduce 
the risks to personnel without compromising mission success. 


Report of the Auditor General of Canada—Fall 2009 


Chapter 5 1 


ACQUIRING MILITARY VEHICLES FOR USE IN AFGHANISTAN 


What we found 
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« In three of the four projects we examined, National Defence and 


PWGSC provided the Canadian Forces in Afghanistan with urgently 
needed vehicles that National Defence determined met the 
operational needs. The quick procurement and delivery of these 
protective vehicles, in the opinion of National Defence, contributed 
to the safeguarding of Canadian soldiers in Afghanistan. National 
Defence and PWGSC successfully implemented some strategies to 
fast-track these three contracting processes, which potentially could 
be applied to streamlining non-urgent acquisitions. For the 

two competitive contracting processes that we examined, we found 
that both processes were managed in compliance with applicable 
contracting policies. However, we found some problems with roles 
and responsibilities and information provided to senior officials. 


The four projects we examined were not managed in accordance 
with National Defence’s project approval guide. Nor did the 
Department have a separate process in place, based for instance on 
the guide, to manage urgent acquisitions to ensure that the projects 
complied with government policies. Most of the required 
documentation was either not prepared or, if prepared, was deficient 
or prepared after the fact. Therefore, the documentation was of little 
value in helping to manage the projects or showing that the Bepiees 
complied with government policies. 


While the Treasury Board of Canada Secretariat reviewed and 
challenged documents submitted by National Defence when it 
sought approval for the projects, some important issues were left 
unchallenged. For example, had officials asked why the required risk 
management strategies were not submitted, they would have learned 
that the strategies had not been prepared—a signal that the project 
management policies may not have been followed. For its part, 
National Defence did not disclose some important information 
needed for informed decision making, which made it difficult 

for the Secretariat to fulfill its challenge function. For instance, 
National Defence did not state that there was a high risk that it 
would not be able to replace all the old tanks in Afghanistan with 
the new Leopard 2 tanks, which was one of the project’s goals. 


The entities have responded. The entities agree with all of our 
recommendations. Their detailed responses follow the 
recommendations throughout the chapter. 
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Introduction 


Canada’s mission in Afghanistan 


5.1. In July 2003, under the auspices of the United Nations, Canada 
assumed command of the International Security Assistance Force 
(ISAF) in Kabul, Afghanistan. Its main focus during the first year of 
command was helping the Afghan Transitional Authority maintain a 
safe and secure environment in Kabul while the country developed 
and ratified a constitution. ISAF became a North Atlantic Treaty 
Organization (NATO) mission in August 2003. 


5.2. In the fall of 2005, the Canadian Forces moved its main 
contingent from Kabul to Kandahar so it could begin operations in 
Kandahar region in March 2006. In doing so, Canada assumed 
responsibility—along with its coalition and Afghan partners—for 
stabilization and security of a larger area. Canada has committed to its 
current role in Afghanistan until 2011. 


§.3 The Canadian Forces recognized that, when it deployed to the 
Kandahar region, its greatest threat would be from land mines and 
improvised explosive devises (IEDs). To counter these threats, the 
Canadian Forces has used increasingly heavily-armoured and blast- 
resistant equipment with the goal of providing better protection 
against IEDs. 


Military acquisitions for use in Afghanistan 


5.4 National Defence has acquired a number of military vehicles on 
an urgently required basis for use in Afghanistan. This is normal as no 
military can acquire equipment to meet all its possible needs. As well, 
the military has to make trade-offs to stay within budgets. Making 
urgent acquisitions can be challenging because military equipment may 
need to be manufactured or, when acquired, modified. In times of 
conflict, allied forces’ demands for similar items may limit availability 
and have an effect on wait times. As a result, windows of opportunity 
are important in deciding what equipment to acquire. 


5.5 In 2005, the Canadian Forces planned to acquire equipment to 
prepare for its operations in Kandahar region as of March 2006. Since 
then, many more projects to acquire a wide range of equipment were 
launched. The equipment has included new protected vehicles, 
improved protection for current vehicles, helicopters, drone aircraft, 
and other systems to improve the safety of soldiers and reduce the 
threat posed by IEDs. 
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5.6 Soserious is the threat from JEDs that in 2008 when the House 
of Commons passed a motion calling for an extension of the Canadian 
military presence in Kandahar to July 2011, several conditions were 
attached, including the acquisition of medium-lift helicopters and 
drone aircraft in order to better protect Canadian troops from IEDs. 


Project management policies and guidelines 


5.7 The Treasury Board’s Policy on the Management of Projects 
requires all federal departments to have sound internal policies, 
guidelines, and practices for the identification, planning, definition, 
approval, and implementation of projects. Departments are to ensure 
that projects follow standard project management principles, such as 
those set out in the Treasury Board’s policies and the Project 


Management Institute’s A Guide to the Project Management Body of 
Knowledge (PMBOK® Guide). 


5.8 National Defence has established its own project management 
framework that includes a Project Approval Guide. This Guide 
provides policy and procedural guidance on the approval process for 
projects, activities, and initiatives that require resource funding— 
including the acquisition of military equipment. The main focus of the 
Guide is to detail the internal decision-making processes to be followed 
and the project management practices to be employed to manage the 
project within approved resources. 


Roles and responsibilities for buying equipment 


5.9 Major defence equipment acquisition projects involve several 
departments. Exhibit 5.1 describes the roles of all the players involved. 


Exhibit 5.1 The roles and responsibilities of federal organizations in acquiring equipment for National Defence 


Organization | . Roles and responsibilities 


Cabinet Ministers’ forum for discussion and decision making. May provide policy approval in principle for some 
major Crown projects. 


Privy Council Office Provides advice on proposals to Cabinet. 


Assists departments in preparing proposals to Cabinet. 


Treasury Board As a statutory committee of Cabinet, approves policies in areas such as project management and 
contracting. 


Decides whether a project should be deemed a Major Crown Project. 


When a project is likely to exceed ministerial expenditure authority limits, approves such projects, allows 
the departments to enter into contracts, and may allow exemptions from Treasury Board policies. 
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Exhibit 5.1 The roles and responsibilities of federal organizations in acquiring equipment for National Defence (cont’d) 


Organization Roles and responsibilities 


Treasury Board of As the Treasury Board’s administrative arm, develops policies and guidance for the federal government on 
Canada Secretariat topics such as project management and contracting. 


Assists departments in preparing submissions to the Treasury Board. 
Analyzes and challenges departmental submissions. 


Provides recommendations and advice to the Treasury Board on project proposals. 


National Defence Initiates and manages acquisition projects. 

Defines project requirements. 

Analyzes and recommends options. 

Proposes a procurement strategy. 

Prepares statements of work and technical bid evaluation plans. 


Performs quality assurance, receives goods, and authorizes payments. 


Public Works and Manages the contracting process. 
Government Services 


Conducts market analyses. 
Canada 


Develops final procurement strategies. 
Prepares bid documents and conducts tendering processes. 


Awards contracts. 


Industry Canada Administers the Industrial Regional Benefit Policy, which provides a framework for linking federal defence 
procurement to long-term industrial and regional development in Canada. This is intended to ensure that 
socio-economic objectives are considered when a major defence acquisition is made. 


Source: Adapted from various Government of Canada documents 


Focus of the audit 


5.10 We conducted the audit to determine whether National Defence 
could demonstrate that the acquisition of four types of military 
vehicles for use in Afghanistan met operational needs, and that the 
vehicles were acquired in a timely manner. We also examined whether 
the contracting processes used by Public Works and Government 
Services Canada and National Defence complied with federal 
government contracting laws and regulations—and with Treasury 
Board policies—while meeting the unique time pressures linked to 
the purchases. 


5.11 In conducting this audit, we recognize that the Canadian Forces 
in Afghanistan is endangered by IEDs and other threats. 

We understand that the acquisitions we have audited were needed as 
quickly as possible to carry out operations and to enhance protection 
of Canadian personnel in Afghanistan. We took these facts into 
account in our assessments of what we found. 
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Project management 


5.12 The audit was limited to four vehicle acquisition projects that 
each cost more than $100 million. Each project was deemed urgent 
by National Defence for the reasons mentioned above. Exhibit 5.2 
describes these four vehicle projects. The audit did not examine 
Industrial and Regional Benefits. 


5.13 More details on the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


National Defence’s project management framework is consistent with Treasury 
Board policies 


5.14 Each of the four acquisition projects we examined’ was to be 
managed in accordance with the Treasury Board’s policies and 
guidelines on capital plans, projects, and procurement. While the 
proposals linked to these purchases clearly conveyed the urgent nature 
of the projects, no exemptions from the Treasury Board’s project 
management policies were sought or provided. 


§.15 The Treasury Board’s project management policies require 
departments to establish sound internal policies, guidelines, and — 
practices. As recommended by these policies, the practices contained 
in the Project Management Institute’s A Guide to the Project 
Management Body of Knowledge (PMBOK® Guide) have been adopted 


by National Defence as its standard for project management. 


§.16 National Defence has established its Project Approval Guide 
(the Guide) as the principal source for policies and procedures on the. 
approval process for projects. The Guide describes in detail the 
program decision documents that staff must prepare to obtain project 
approvals. It also provides some guidance on the practices to be 
employed for managing projects. 


5.17 We reviewed the contents of the Guide against the requirements 
contained in the Treasury Board’s project management policies and 
found that they were consistent. We found that there was no separate 
process in the Guide to deal with urgent operational requirements 
(known as UORs), nor is there a specific project management 
framework for UORs. As a result, the Guide was the only source of 
criteria available to us in auditing these urgent projects. Based on this, 
we expected the Department, for each of the projects we audited, to 
have complied with the requirements of the Guide or to have senior 
management approve and document specific deviations. 
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Exhibit 5.2 Four types of vehicle acquisitions were audited 


Armoured Patrol Vehicle - 
- (RG-31) 


RG-31s in a convoy in Afghanistan 


Date of project approval: 
October 2005 


Approved cost: $175 million 


The purpose of this purchase 
was to provide the Canadian 
Forces with a better-protected 
patrol vehicle when it began 
operations in Kandahar region 
. in February 2006. At the 
time, this purchase was the 
top priority of the Canadian 
Forces. 


One of National Defence’s 
stated reasons for the project 
was to replace the G-Wagon— 
a small patrol vehicle that did 
not provide enough protection 
to personnel in Afghanistan. 
Another was to replace the 
Bison, a larger vehicle used to 
transport personnel. 


In buying the RG-31, the 
Department got a vehicle that 
includes a remote weapon 
station (RWS). This allows an 
operator to fire weapons from 
within the vehicle—using a 
monitor and joystick—while 
remaining protected. 


The Department bought 

75 RG-31s under contract 
with General Dynamics Land 
Systems Canada (GDLS- 
Canada). They have been 
used in Afghanistan and for 
training in Canada. 


Source: Department of National Defence 


Light Armoured Vehicle 
Remote Weapon Station 
(LAV RWS) 


~ = WaMn ey 


LAV RWS 


Dates of project approval: 
October 2006, March 2007, 
and April 2007 


Approved cost: $118 million 


The purpose of this purchase 
was to produce a variation of 
the Canadian Forces’ Light 
Armoured Vehicle III (LAV III) 
that would provide better 
protection against mines and 
IEDs. The LAV RWS would be 
used in the same manner as 
the existing LAV Infantry 
Section Carrier. 


Two key features of the new 
vehicle are a remote weapon 
station (RWS) and custom 
armour protection. Instead of 
the turret mounted on the 
standard LAV, the RWS 
reduces the vehicle’s weight 
by several thousand pounds, 
which allows additional 
armour to be installed. Never- 
used chassis from the original 
LAVs are being used as the 
basis for producing 33 of the 
LAV with RWS. 


Under contracts with General 
Dynamics Land Systems 
Canada (GDLS-Canada) and 
Armatec, the vehicles are 
currently being produced for 
delivery in 2009. 


U 


Armoured Heavy Support 
Vehicle System (AHSVS) 


AHSVS transporting a-damaged tank 


Date of project approval: 
October 2006 


Approved cost: $169 million 


The purpose of this purchase 
was to replace the Heavy 
Logistic Wheeled Vehicle that 
was being used in Afghanistan 
with a vehicle better protected 
against the threat of 
improvised explosive devices 
(IEDs). It is mainly used to 
transport supplies and 
equipment. ~ 


Ninety-five vehicles of various 
types, and related equipment, 
have been acquired under — 
contract with DaimlerChrysler 
AG. These are currently used 
for operations in Afghanistan 
and for training in Canada. 
Two additional vehicles are 
still to be received as 
compensation for delivery 
delays. 


Leopard 2 Tank 


Leopard 2 Tank 
Date of project approval: 
March 2007 


Approved cost: $650 million 


The first goal was to lease 

20 Leopard 2 tanks and two 
special-purpose armoured 
vehicles from Germany in 
order to meet the urgent need 
to replace the Leopard 1 tanks 
that had been deployed to 
Afghanistan. 


The second goal was to 
purchase 100 Leopard 2 
tanks, including special- 
purpose armoured vehicles 
from the Netherlands, to 
provide immediate training 
vehicles for troops in Canada 
and a longer-term tank 
capability. 


Twenty tanks were borrowed 
(not leased as originally 
approved by the government) 
from Germany. These arrived 
in Afghanistan, for the most 
part, in late 2007. 


One hundred tanks were 
purchased. Under a separate 
authority, National Defence 
purchased an additional 

15 used tanks that it intends 
to use for spare parts. The 
project is still in progress. 
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5.18 National Defence’s Guide requires projects to be managed in 
phases, with key decision points along the way. Compliance with the 
Guide requires that many steps and key documents—meeting both 
Treasury Board requirements and National Defence’s internal 
decision-making needs—be completed. Projects at National Defence 
begin with a clear statement and agreement by senior management on 
the capability deficiency to be resolved. Once this is clearly 
understood, options are to be identified and analyzed, and an informed 
decision made. Based on this decision, the project is supposed to be 
planned in detail before it is implemented. 


National Defence did not meet its own project approval guidelines for these 
acquisitions 


5.19 As outlined in Exhibit 5.3, we found that for the four projects 
we audited, most of the required documents were not prepared or, if 
prepared, they were either deficient or prepared after the fact and thus 
of little value in helping to manage the projects or in showing that 
project management met Treasury Board requirements. In such cases, 
National Defence did not document why the required documentation 
was not prepared. 


Exhibit 5.3 National Defence did not meet most requirements of its Project Approval Guide when it acquired urgently needed equipment 


Capability Deficiency 


Identification Phase 


Light 
Armoured 
Armoured _ Vehicle 
| Heavy Support Remote 
Armoured Vehicle Weapon 
: Patrol Vehicle System Station Leopard 2 
Required Document — Purpose (RG-31) (AHSVS) (LAVRWS) | — Tank 


Provides a clear and agreed-upon statement 


of the capability deficiency to be resolved by an) oy) Cy wz 
the project. 
Broad Options Identifies a broad range of potential solutions. 
Analysis O O ® 
Identification Phase Gives internal approval to proceed with : 
Approval planning and provides the total anticipated N/A @ @ @ 


project cost. 


; Options Analysis Phase . 


Project Charter 


A mandatory project document to be 
completed as early as possible, the charter 
establishes the mandate for how the project 


will be organized. 
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Exhibit 5.3 National Defence did not meet most requirements of its Project Approval Guide when it acquired urgently needed equipment (cont'd) 


Light 
Armoured 
Armoured Vehicle 
Heavy Support Remote 
Armoured Vehicle Weapon 
Patrol Vehicle System Station Leopard 2 
Required Document Purpose (RG-31) ~ (AHSVS) (LAV RWS) Tank — 
Options Analysis Provides a detailed comparison including 
costs of alternatives. o © O O 
Project Profile and Outlines the risks associated with a project 
Risk Assessment and the management strategies to deal with 
them. It is to be provided to the Treasury @ @ O @ 
Board of Canada Secretariat in advance of 
making submissions. 
Preliminary Project Allows the Department to obtain internal 
Approval approval in principle for the preferred option 
at the estimated cost and to proceed to the Nes ie Mis O 
next phase. 
Definition Phase 
Statement of Creates a statement of the characteristics 
Operational that must be delivered for the project to = co O o 
Requirement satisfy the capability deficiency. 
Substantive Cost This is a quality cost estimate that takes into 
Estimate account all project objectives and @ - € 6 
deliverables. 
Project Management | Defines how the project is to be executed, 
Plan — monitored, controlled, and closed. O O O O 
Effective Project Grants internal approval for the selected 
Approval option and allows the project to proceed to cs @ @ C) 
the next phase. 
Implementation Phase 
Initial Operational Shows attainment of the minimum ability to 
N/A 
Capability Certificate | effectively employ a capability. © O , O 
Full Operational Shows full attainment of the ability to 
#3 N N/A N/A 
Capability Certificate | effectively employ a capability. 6 te ; : 
Input into the Throughout the project, key information is to 
Capability be entered into this database. oo = & 8 


Investment Database 


2) Met the Guide’s requirements 


— Met some of the Guide’s requirements 


C) Required document was not prepared or had significant deficiencies 


N/A—Does not apply or is not yet required 


Source: The list of documents required and their purpose was adapted from the Project Approval Guide, Department of National Defence. The assessments were 
prepared by the Office of the Auditor General of Canada. 
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5.20 The problem National Defence faced is that its Guide is geared 
to projects that develop over time, in phases. The four projects we 
examined simply did not progress in clear stages. Instead, their phases 
tended to be completed all at once or were compressed into a short 
period of time. Because the planning of these projects was done so 
quickly, officials told us that they could simply not prepare all the 
documentation required by the Guide and also get the job done. In the 
absence of a framework for dealing with UORs, they were left to decide 
on their own what documents to prepare. 


5.21 The Department has recognized the need to focus on UORs 
and, in 2008, it established a UOR policy. However, that policy only 
describes who is responsible for identifying these needs within National 
Defence. It does not set out a specific project management framework 
for UORs. Such a framework is important for the following reasons: 


¢ First, a framework will ensure that National Defence is able to 
demonstrate managerial oversight and decision making. In our 
audit work, we found that key advisory and decision-making bodies 
like the Senior Review Board and the Program Management Board 
either did not provide advice and make decisions as required under 
the Guide or made them much later, after the key decisions had 
been made. In addition, we found instances where the internal 
document used to inform decision makers, known as the synopsis 
sheet, was either not prepared or lacked important information for 
decision making. While it is clear that decisions were made and 
that, based on those decisions, actions were taken, the Department 
had not documented the decisions and its rationale in the 
expected manner. The Department informed us that, since the 
time of these projects, it has made changes to how such decision- 
making documents are prepared, with the goal of using a common 
document for both internal decisions and Treasury Board 
submissions, as much as possible. 


¢ Second, a framework will help ensure that sound project 
management principles are applied and that Treasury Board 
policies are followed. For example, with the tank replacement 
project, plans were prepared on the basis that the Leopard 2 tanks 
to be purchased would be fitted with important implements such 
as mine ploughs and bulldozer blades. No research was done to 
find out if these implements could actually be fitted on the tanks. 
Later, it was determined that it would be more difficult and 
time-consuming than originally expected to install them on the 
purchased tanks. As a result, National Defence has now 
transferred this requirement into a related project—the Force 
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Mobility Enhancement Project. This project, estimated at 

$376 million, will also provide special-purpose Leopard 2 
armoured vehicles that National Defence could not afford within 
the funding approved for the Tank Replacement Project. 


¢ Finally, a framework will help National Defence demonstrate that 
it has chosen the right piece of equipment to address the 
capability gap it is trying to fill. The Guide and Treasury Board 
policies require all projects to conduct an options analysis with a 
thorough analysis of the preferred option. This allows for informed 
decision making. It also allows those who are making decisions to 
consider trade-offs among features such as performance, 
complexity to implement, and cost. We found deficiencies of 
varying degrees in the options analysis for all four projects. 


§.22 Recommendation. National Defence, in consultation with the 
Treasury Board of Canada Secretariat, should examine its Project 
Approval Guide to determine how the Department can manage the 
process of acquiring urgently needed equipment in a timely manner 
while respecting sound project management principles and Treasury 
Board policies. 3 


National Defence’s response. Agreed. National Defence will review 
the current project approval guidance with a view to align the present 
documentation requirements for procurement greater than $1 million 
with the project approval framework as outlined in the Project 
Approval Guide. The Treasury Board of Canada Secretariat and other 
stakeholders will be consulted in order to achieve the correct balance 
between urgency and appropriate levels of documentation to support 
decision-making rationale. The National Defence group responsible 
for materiel will be consulted to ensure that coordination of the Guide 
and Project Management Manual meets the requirements. 


The contracting processes we examined complied with government policies 


5.23. The purchase of military equipment must be carried out in 
compliance with all laws, regulations, and Treasury Board policies that 
apply to government purchases. Public Works and Government Services 
Canada (PWGSC) maintains what is known as the Supply Manual, 
which describes and guides such activities. It contains relevant laws and 
regulations, as well as government and department policies. 


5.24. While PWGSC is the contracting authority, it shares 
responsibility for some aspects of the acquisition process with National 
Defence. The way responsibilities are divided is set out in a 
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memorandum of understanding (MOU) between the two departments 
and published in the Supply Manual. 


5.25 Compliance with contracting laws, regulations, and policies is 
essential. It helps ensure that acquisitions meet real needs, provide the 
best value for money, promote competition (with some exceptions), 
ensure fair and equal treatment of potential suppliers, and maintain 
the accountability and integrity of the contracting process. While a 
number of acquisition strategies are available under government 
policies, a competitive process is deemed to be the best way to achieve 
these goals. 


5.26 Our audit examined the Armoured Patrol Vehicle (RG-31) and 
the Armoured Heavy Support Vehicle System (AHSVS) acquisitions, 
both managed as competitive projects, to determine whether PWGSC 
and National Defence had managed them in compliance with applicable 
contracting policies. To determine this, we reviewed PWGSC 
acquisition files and National Defence project files. We also conducted 
interviews with department officials and some company representatives. 


5.27 While we found a few problems with the contracting processes 
for these vehicles, overall both processes were managed in a way that 
complied with the applicable contracting policies. As well, the 
contracting processes were completed in a timely manner. 


Roles and responsibilities related to contracting for the Armoured Patrol Vehicle 
(RG-31) were not followed 


’ 


§.28 During the summer of 2005, to prepare for the start of operations 
in the Kandahar region as of February 2006, National Defence 
assembled a list of critical equipment it needed to buy. The most 
important item on this list was an Armoured Patrol Vehicle (RG-31) to 
carry out operations while protecting troops against improvised 
explosive devices (IEDs) and land mines. Analysis prepared by the 
Canadian Forces showed that without this vehicle, the lives of 
Canadian soldiers would be at extreme risk or the scope of operations 
would need to be adjusted. 


5.29 In August 2005, National Defence began to contact companies 
to obtain technical specifications, price, and delivery times of vehicles 
on the market. We found that National Defence informed one 
potential supplier, which responded to the survey, that its product 
would not meet the Department’s need. The supplier complained that 
if National Defence had clearly stated its requirements, it would not 
have spent time and effort to provide National Defence with 
information about its product. 
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5.30 Based on the MOU between National Defence and PWGSC 
(described in paragraph 5.24), this type of survey is not National 
Defence’s responsibility. Even though National Defence contacted 
PWGSC in mid-August, it nonetheless continued to conduct 

its survey. 


5.31 In early September 2005, National Defence decided it wanted to 
sole-source the purchase of 50 vehicles called the RG-31. The 
Department believed that the RG-31 would meet its needs and that, 
based on the information it had collected, only one company could 
meet its essential technical requirements and deliver quickly. 


5.32 According to Department documents, in September 2005, the 
supplier of the RG-31 advised National Defence that it had held a 
place for a Canadian order and could begin delivery in March 2006 but 
the contract had to be signed by late October 2005. After that date, 
according to these documents, the company told the Department that 
the RG-31 might not be available within this time frame due to orders 
from other militaries. National Defence informed PWGSC of these 
facts. It also informed PWGSC that it preferred to use a sole-source 
contract for this acquisition. PWGSC believed that a competitive 
process could fulfill National Defence’s urgent requirement. 
Nonetheless, it asked National Defence to present its reasons for 
requesting an exception. Under the federal government’s contracting 
regulations, competitive bids must be sought before a contract is 
entered into, unless the project meets one of the reasons for an 
exemption set out in the regulations (Exhibit 5.4). Although National 
Defence was concerned that a competitive process might cause it to 
lose a chance to acquire a suitable vehicle that the supplier could begin 
delivering as early as March 2006, the beginning of operations in 
Kandahar, it did not provide the rationale for a sole-source contract. 


Exhibit 5.4 General reasons for exceptions to competitive processes 


Before any contract is entered into, the contracting authority shall solicit bids. However, 
a contracting authority may enter into a sole-source contract without soliciting bids 
where any one of the following criteria is met: 


¢ The need is one of pressing emergency in which delay would be injurious to the 
public interest. 


* The estimated expenditure does not exceed certain limits depending on the type of 
acquisition. 


¢ The nature of the work is such that it would not be in the public interest to solicit bids. 


¢ Only one company is capable of performing the contract. 


Source: Government Contracts Regulations 


Report of the Auditor General of Canada—Fall 2009 


Chapter 5 13 


ACQUIRING MILITARY VEHICLES FOR USE IN AFGHANISTAN 


14 


Chapter 5 


5.33 Before it could decide on the type of contracting strategy to 
pursue, PWGSC needed to assess the competitive environment for 
armoured patrol vehicles. It could not rely on the informal price and 
availability survey completed by National Defence, so it invited 
interested suppliers to respond to its own price and availability request. 
Among other things, it asked suppliers if they could meet the stated 
requirement of delivering the first 25 vehicles within seven months 
after the placement of orders—which we calculate would be 

May 2006, at the earliest. Based on its assessment, PWGSC found that 
three firms would be able to meet the technical requirements set out by 
National Defence and that a competitive environment existed. 
However, the ability of suppliers to meet the seven-month delivery 
requirement was not taken into account at this stage. Our review of 
PWGSC’s documentation found that only one supplier showed that it 
could meet the required delivery schedule. In our opinion, because the 
purchase was urgently needed, the ability of potential suppliers to 
deliver vehicles to Afghanistan in accordance with this key 
requirement should have been taken into account at that stage. 


5.34 At the beginning of October 2005, the government granted 
PWGSC advance authority to contract for 50 armoured patrol 
vehicles. This authority did not specify whether the contract had to be 
competitive or sole-sourced. 


5.35 The three companies that PWGSC assessed as meeting the 
technical requirements set out in the price and availability request 
were invited to prepare bids. PWGSC developed a compressed 
competitive procurement schedule showing that a contract with the 
successful bidder be awarded at the end of October 2005. This was 
done to meet National Defence’s goal that an order be placed for 
armoured patrol vehicles, by late October 2005 so that it would not 
miss the opportunity to purchase the RG-31, should it be determined 
the best vehicle. 


5.36 Due to the compressed schedule, PWGSC was unable to prepare 
many of the required documents before it issued the request for 
proposal (RFP). PWGSC notified the bidders that these documents 
would be provided in phases and finalized them during the bidding 
period. In fact, some changes were still being made a few days before the 
end of the bidding period, based on new information National Defence 
was giving PWGSC. Normally, this documentation is ready before 

the competition begins. All of this had an impact. For example, the 
weighting related to the requirements was provided to the companies 
midway in the bidding process. At that point, the companies learned 
that the weighting for delivery favoured delivery beginning in 


Report of the Auditor General of Canada—Fall 2009 


ACQUIRING MILITARY VEHICLES FOR USE IN AFGHANISTAN 


February 2006. The two companies that did not bid expressed concerns 
about changes made during the competitive process, although one of 
them only communicated its concerns on the day of bid closing. One of 
the companies stated that since the weighting favoured earlier delivery, 
it would not be able to compete. It felt that this weighting did not 
reflect its understanding that according to PWGSC’s price and 
availability request, delivery of vehicles would not be required to begin 
until seven months after receipt of order. 


5.37 At the end of the bidding process, only the vendor of the RG-31 
submitted a bid. However, PWGSC and National Defence determined 
that the bid was non-compliant. Based on PWGSC policies, when no 

compliant bids are received, the competition should be cancelled and 

re-tendered. 


5.38 PWGSC cancelled the competitive process but did not consider 
re-tendering to be an option because, according to documents, it 
would have caused an unacceptable delay that might have 
compromised mission success and personnel safety. PWGSC and 
National Defence conducted a clarification process with the bidder 
and concluded that the proposal met all the requirements to their 
satisfaction. As a result, PWGSC decided to sign the contract based on 
government contracting regulations, which state that when the need is 
“one of pressing emergency in which delay would be injurious to the 
public interest” (Exhibit 5.4), it can issue a sole-source contract. 

The contract was signed on 31 October 2005. 


5.39 This process successfully provided National Defence with the 
vehicle it sought, in a timely manner. However, time was wasted when 
National Defence failed to involve PWGSC early in the process and 
PWGSC had to conduct another price and availability request. 
Although PWGSC promoted competition throughout the acquisition 
process, an exception under the government’s contracting regulations 
had to be used to finalize it. 


5.40 Recommendation. For urgent operational requirements, 
National Defence should involve Public Works and Government 
Services Canada (PWGSC) early, in accordance with agreed upon 
roles and responsibilities, so that effective procurement strategies can 
be implemented to meet operational objectives. 


National Defence’s response. Agreed. Lessons learned from recent 
procurements, and opportunities for making process and other 
improvements, are being explored in some depth as part of a number of 
ongoing government reviews of defence procurement. Measures to 
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further improve the already close collaboration between National 
Defence and PWGSC, while respecting the statutory responsibilities of 
the respective departments, are being examined as part of these 


reviews. 


In addition, PWGSC Acquisitions Branch now has full membership in 
several standing and ad hoc National Defence bodies that oversee the 
planning of future acquisitions, both routine and urgent. 


Incomplete information was provided to senior officials on the AHSVS contracting 


5.41 In October 2006, based on a competitive procurement strategy, 
the government granted the Minister of Public Works and Government 
Services Canada (PWGSC) advance authority to enter into a contract 
to buy 82 armoured heavy support vehicle systems (AHSVS) of various 
types, as well as related support and an option to acquire additional 
vehicles and equipment, for a total cost of $169 million. The contract 
was signed approximately six months later on 30 March 2007. 


5.42 PWGSC published a notice that the Government of Canada had 
a requirement for AHSVSs and companies could obtain a copy of the 
price and availability request. The price and availability requests that 
were completed by interested companies helped National Defence set 
its requirements and helped PWGSC develop and issue a solicitation 
of interest and qualification (SOIQ). According to the procurement 
strategy outlined in PWGSC’s request to government asking for 
advance authority to enter into a contract, only suppliers prequalified 
using the SOIQ were to be sent a formal RFR PWGSC informed us that 
the SOIQ was intended to evaluate potential suppliers against a set of 
63 high-level mandatory protection and performance requirements 
whereas the RFP would include the full set of requirements. However, 
none of the participating companies demonstrated that they could meet 
the SOIQ requirements. At this point, the original procurement plan 
could not be followed. This was a very serious situation because 
PWGSC would not be able to proceed with the purchase of vehicles 
that National Defence said it urgently needed to protect its troops in 
Afghanistan against IEDs. According to PWGSC’s supply manual, 
when there is a significant change in the procurement strategy, the 
procurement plan should be revised and approved at the Assistant 
Deputy Minister (ADM) level. In our opinion, at this point the plan 
should have been revised. However, PWGSC maintains that the result 
of the SOIQ did not represent a change in the competitive 
procurement strategy approved by the government, and therefore 
ADM-level approval of a revised procurement plan was not required, 
since a competitive process was still conducted. 
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5.43 Instead, PWGSC and National Defence officials visited all of 

the suppliers who had responded to the SOIQ, and some that had not. 
The visits were conducted, with a view, among other things, 

to reassessing the requirements for the vehicles based on vehicles the 
companies could actually supply. The mandatory requirements of the 
SOIQ were modified and a total of 172 were set out in the RFP 
National Defence informed us that changes to the mandatory 
requirements were partially based on information it had collected during 
its visits with companies, and that it believed these requirements could 
be met. However, we were unable to identify what changes were made to 
the requirements as a result of the field visits because National Defence 
did not prepare detailed records of the visits. PWGSC issued an RFP to 
six companies. Although three of them submitted complete proposals, 
only one was able to meet all the requirements. According to PWGSC’s 
evaluation documents, the two other bidders did not demonstrate that 
they could meet a significant number of the mandatory requirements. 


5.44 When the Minister of Public Works and Government Services 
Canada was asked to authorize the issuance of the contract to the 
winner of this competition, the memorandum outlined, among other 
things, highlights of the contracting process. It clearly stated that 

“the award is in accordance with the strategy outlined in the submission” 
that was approved by the government. In our opinion, this 
memorandum, rather than stating that the award was in accordance 
with the procurement strategy, should have informed the Minister that 
there had been a significant departure from the strategy when no 
companies qualified during the SOIQ phase. PWGSC has acknowledged 
that it could have provided more details in the memorandum but, in its 
opinion, it complied with the overall procurement strategy, which was to 
have a competitive procurement process. 


5.45 In addition, the government required PWGSC to report to the 
Treasury Board of Canada Secretariat after the contract was awarded. 
This report was to contain information that included confirmation 
that the acquisition was carried out in accordance with the 
procurement strategy as outlined in government documents. We found 
that PWGSC did not specifically state whether or not the acquisition 
was carried out in accordance with the procurement strategy and did 
not disclose that the procurement strategy was modified when none of 
the companies was found compliant with the SOIQ’s requirements. 
In our opinion, PWGSC should have disclosed this information. 
PWGSC informed us that more detailed information could have been 


provided in its report. 
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5.46 We also found some problems with the way these contracts were 
administered. We found that National Defence accepted delivery of a 
truck not included in the original contract. This should not have 
happened until the contract was amended in writing. Doing so was not 
in accordance with the government’s contracting policy. We also found 
that National Defence paid twice for the same vehicle and that this 
error was not discovered until two months later when the supplier 
notified the Department of this fact. National Defence has informed us 
that the amount paid in error has since been credited to the 
Department’s account with the vendor. 


PWGSC and National Defence worked together to meet urgent needs 


5.47 We expected National Defence and PWGSC to work together 
to ensure effective contracting as they are responsible for the 
procurement process. We found that PWGSC and National Defence 
formed integrated teams dedicated to administering the contracting 
process. Documents presented to government were prepared jointly. 
Our review of files showed that the two departments worked quickly 
and together to resolve problems when procurement strategies were 
not proceeding as planned. They cooperated to ensure that the 
allocation of PWGSC’s limited staff reflected the priorities of National 
Defence. We believe that there are some positive lessons to be learned 
from the way in which the two departments worked together that 
could be applied to the regular contracting process. 


§.48 Recommendation. Public Works and Government Services 
Canada (PWGSC) and National Defence should examine lessons 
learned in contracting for urgent operational requirements that could 
be applied to help speed up the regular procurement process. 


National Defence’s response. Agreed. National Defence continues to 
work with PWGSC and other key partners to develop a 
whole-of-government approach to improve and accelerate military 
procurement. Lessons learned from recent projects related to urgent 
operational requirements are being evaluated and contributing to 
these ongoing efforts. In particular, National Defence is working with 
PWGSC to develop a comprehensive defence procurement process 
model that will help optimize the process for both routine and urgent 
military acquisitions. To this end, National Defence has largely 
completed the initial mapping of its current internal processes and 


plans to have the full National Defence model completed by 
31 March 2010. 
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PWGSC’s response. Agreed. The observations noted in the chapter 
regarding the implementation of strategies to fast track the 
procurement and delivery of the urgently required military vehicles, 
while respecting government contracting policies and objectives, 
reflect the success of these PWGSC military procurement initiatives. 


PWGSC will conduct a lessons-learned exercise to review the 
procurement activities undertaken during the urgent acquisition of the 
four types of military vehicles destined for Afghanistan, examined 
during this audit. The lessons learned will highlight the steps, 
activities, actions, and processes that led to the quick and timely 
acquisition of these vehicles. 


The lessons learned will be completed by the end of October 2009 and, 
where applicable, effective practices identified through the 
lessons-learned exercises would be considered for implementation as 
part of the Department’s existing initiative aimed at improving the 
efficiency of major military and other Government of Canada 
procurements. 


549 We examined whether National Defence could demonstrate that 
the four types of vehicles that were subjects of our audit 


* adequately addressed the capability deficiencies identified 
by the Department, 


* met the Department’s performance expectations, and 


¢ could be properly maintained in Afghanistan. 


The armoured patrol vehicles (RG-31s) are meeting operational needs 


5.50 National Defence’s reasons for purchasing the Armoured Patrol 
Vehicle (RG-31) focused on solving two problems: 


* replacing the G- Wagon because it was inadequately protected 
against the threats that soldiers faced in Afghanistan, and 


* providing protected transport for personnel. 


5.51 Based on National Defence’s assessments, the mine-resistant 
vehicle it bought (the RG-31) is saving lives. When the vehicle was 
first deployed in Afghanistan, it had mechanical problems and a 
shortage of spare parts that affected its availability. We found that 
these problems had been resolved. In February 2008, the Chief of Land 
Staff certified that the project met its full operational capability, with 
the exception of some related equipment that was yet to be purchased. 
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Serviceability—The number of pieces of 
equipment ready to be used for their intended 
purpose from among those available. An item is 
not serviceable if it needs maintenance or repair. 
For example, if 10 trucks were available, of 
which 7 could be put into service and 3 were 
undergoing repair, the serviceability rate would 
be 70 percent. 
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5.52 Normally, equipment is tested before it is used on deployments. 
But due to National Defence’s urgent need, the RG-31s were delivered 
directly to Afghanistan without normal testing. The mission quickly 
found several safety and performance shortfalls and the Department 
corrected these problems. 


5.53 We also found that the initial purchase of 50 RG-31s provided 
National Defence with fewer vehicles than it needed for operations. 
When we compared the number of RG-31s that it bought to the 
detailed military analysis of the number needed—including those 
needed for training—we found that the number needed was more than 
the number the Department purchased. None of the departmental 
decision documents we reviewed explained the reasons for this 
difference. However, we were informed that this difference was due to 
funding limits. 


5.54 The contract included an option to buy 25 more vehicles, based 
on National Defence’s estimate of potential future losses. In May 2006, 
the Department exercised this option and these vehicles were 
immediately placed into operations. The total number of vehicles was 
still less than the number shown as required in the original military 
analysis. In addition, RG-31 losses are not being replaced. 


5.55 We found that the RG-31s were delivered to Afghanistan before 
the target date contained in the Department’s documents to the 
government. Even though the acquisition was made quickly, it was not 
possible to have full delivery in Kandahar before the start of operations 
in March 2006. Consequently, National Defence considered short- 
term options, including borrowing RG-31s from Canada’s allies. 
National Defence decided to continue using its G-Wagons with 
additional armour protection as an interim measure. 


5.56 Even though the RG-31 has had a few problems, National 
Defence has concluded that the vehicle met the early operational 
needs of the mission. 


The AHSVS is meeting operational needs 


5.57 The Armoured Heavy Support Vehicle System (AHSVS) has 
successfully met the Department’s goal of replacing its heavy logistic 
wheeled vehicles. The Commander of Task Force Afghanistan 
reported internally that the AHSVS has been well received. 

Our analysis of data found that the AHSVS has had a high 


serviceability rate—higher than the vehicle it replaced. 
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5.58 We found that the AHSVS project delivered vehicles to 
Afghanistan about six months later than the original target date. 

A number of factors contributed to this delay. First, the contract was 
entered into two months later than planned because the procurement 
strategy took longer than expected. Second, the negotiated delivery 
schedule in the contract was not aligned with the original planned 
schedule. Third, production was delayed because a sub-contractor that 
produced the armoured cab portion of the truck encountered a 
shortage of steel. To compensate for this production delay, the supplier 
will provide National Defence with two additional vehicles. Our audit 
did not identify any significant impact on operations as a result of 
these delays. 


§.59 The Canadian Forces informed us that the AHSVS project has 
achieved its overall intent of replacing almost all of the heavy logistic 
wheeled vehicles used outside the protection of the military bases 

in Afghanistan. | 


§.60 While not a stated capability deficiency that the AHSVS project 
was expected to resolve, National Defence acquired passenger-carrying 
modules for the AHSVS to help resolve a long-standing need to 
transport personnel in Afghanistan using protected vehicles. These 
were requested a month after the original contract was signed, by 
exercising an option in the contract. However, the availability of the 
passenger modules was limited at first by serviceability issues. 

The Department informed us that these issues are now resolved. 


§.61 The transportation of personnel between sites was not a new 
issue. The Department had sought government approval three times to 
resolve the problem. It should have been addressed by the Armoured 
Patrol Vehicle (RG-31) project. Based on a document from National 
Defence to government, the issue was also supposed to be addressed by 
the Light Armoured Vehicle Remote Weapon Station (LAV RWS) 
project rather than the AHSVS project, which ultimately acquired 
passenger-carrying modules. Finally, the Army now plans to use 
vehicles bought as part of another project to provide troop transport. 
In fact, none of the projects examined in this audit entirely fulfilled 
this capability gap as expected. 


The Tank Replacement Project did not meet all of the most urgent needs 


5.62 The Tank Replacement Project consists of two parts. The first 
part was the most urgent. It was to borrow 20 Leopard 2 tanks and 

2 armoured special purpose vehicles from Germany to replace all the 
old Leopard 1 tanks being used by the Canadian Forces in 
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Afghanistan. The second part was intended to equip the Army with a 
fleet of up to 82 Leopard 2 tanks and 18 special purpose armoured 
vehicles. The Leopard 2 is a modern tank that provides the Canadian 
Forces in Afghanistan with a high level of protection from land mines 
and improvised explosive devises (IEDs). It also has a powerful direct 
fire capability. 


5.63 Based on the Department’s assessments, the borrowed Leopard 2 
tanks have been deployed to Afghanistan and are saving lives. 
However, we found that the tanks have not replaced all of the older 
Leopard | tanks in Afghanistan because the Leopard 2 tanks cannot 
be fitted with the Department’s land mine ploughs, bulldozer blades, 
and mine rollers. These implements are essential to overcome 
obstacles and mitigate the threat posed by IEDs. As a result, the 
mission in Afghanistan continues to use several of the Leopard | tanks. 


5.64 Inearly 2008, the Department began work to find a way of fitting 
mine rollers to the borrowed Leopard 2 tanks. We were informed that 
_ work on this issue is ongoing. 


§.65 Our audit found that soon after deployment, a portion of 

the new tank fleet could not be used due to equipment failure. 

This problem was compounded by a shortage of parts. While normally 
considered a last resort, Task Force Afghanistan almost immediately 
began taking parts from tanks on site in Afghanistan and from tanks 
that National Defence had bought for troop training in Canada in 
order to make the necessary repairs. Despite these challenges, the 
Canadian Expeditionary Command informed us that it is satisfied with 
the level of serviceability of this equipment. 


5.66 The stated goal of the tank replacement project was to have 

all the borrowed tanks ready for use in Afghanistan by the end of 
July 2007. We found that National Defence was unable to meet this 
goal and it was not until the end of August 2007 that the first 

four tanks and two special-purpose armoured vehicles were delivered 
to Afghanistan. Not until mid-October 2007 was the Canadian 
Expeditionary Command satisfied that everything was in place to 
permit their use. Most of the remaining tanks became available 

for operations in Afghanistan by the end of December 2007. 

The Canadian Expeditionary Command informed us that, despite the 
urgent need to have the tanks available by July 2007, it was not aware 
of any negative impact on operations when this goal was not met. 
One of the reasons for this was that the operation was able to continue 
using the old Leopard 1 tanks. 
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9.67 It should be noted that while there had been slippages in the 
project, senior officials recognized from the outset that the schedule 
was ambitious and acknowledged that an extraordinary amount of 
work had been accomplished in a short period of time by all involved. 


5.68 One of the reasons National Defence used in support of its bid to 
purchase 100 Leopard 2 tanks from the Netherlands was to have 20 of 
these tanks immediately available to train soldiers in Canada, as they 
were similar to the borrowed tanks being used by the Canadian Forces 
in Afghanistan. However, these training tanks will remain in Europe to 
be swapped for the borrowed German tanks that would otherwise have 
been taken from Afghanistan and returned to Germany by 

September 2009. These tanks will need to be modified to the same 
specifications as the borrowed tanks and since these have been 
cannibalized for spare parts, those parts will need to be replaced while 
the modifications are taking place. These actions will provide Canada 
with Leopard 2 tanks for use in operations until the recently purchased 
tanks are upgraded. However, as a result, Canadian soldiers are being 
sent to train with the German army, creating additional costs and 
inefficiencies. In addition, because of the cap on funding for upgrading 
the Leopard 2 tanks, National Defence will not be able to upgrade all 
the tanks to the desired level. The result is that the training fleet will 
consist of a different model of tank than the deployment fleet—a 
situation that the Canadian Forces considers less than ideal. 


5.69 We found that National Defence now expects to be able to fully 
employ the purchased tanks almost two years later than originally 
planned. This is due to, among other things, delays in conducting a 
feasibility study to help plan the upgrades to the new tanks. 
Documentation that we obtained from Public Works and Government 
Services Canada (PWGSC) shows that it issued and then cancelled a 
request for proposal (RFP), thus preventing National Defence from 
entering into a contract with the Original Equipment Manufacturer 
(OEM) to gather the information required for planning the 
modifications. The reason documented by PWGSC was that this 
would prevent the OEM from gaining any advantage in subsequent 
contracting processes related to the tanks. National Defence also 
informed us that PWGSC prevented it from contacting the OEM. 

As noted earlier, the project will not be installing implements on the 
tanks. A new project has begun, which will include the requirement 


to deliver this capability. 
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The LAV RWS project planning underestimated training requirements 


5.70 When the LAV RWS project was approved, National Defence 
did not know the number of vehicles that would ultimately be 
available for operations in Afghanistan, as training requirements had 
not been adequately taken into account. Project planning documents 
show that the Department initially believed that because the new 
vehicle was so similar to the LAV III, troops would need only minimal 
additional training to use the new vehicle. The planning documents 
did not indicate how many LAV RWS, if any, would be required for 
training. Iwo years later it became clear to the Department that a 
higher degree of training was needed. Consequently, of the 33 vehicles 
being built, 15 will be kept in Canada for training. 


5.71 The original project schedule stated that the vehicles would be 
ready for use in Afghanistan by February 2008. However, there have 
been delays. Compared to the three other projects we audited, this 
project is taking the longest to deliver (Exhibit 5.5). As of June 2009, 
this project has yet to provide vehicles to Afghanistan. The Department 
now plans to field the vehicles sometime later in 2009. We found that 
the delay is the result of the complexity of this project. This has had an 
impact on the Canadian Forces’ training plans and armoured vehicle 
fleet management for the mission in Afghanistan. 


Exhibit 5.5 Project timelines have varied considerably 


Time from government approval to date first vehicle 
Type of Vehicle ee was available for use in Afghanistan 


Armoured Patrol Vehicle (RG-31) 5 months 


Leopard 2 Tank 7 months 


Armoured Heavy Support Vehicle 16 months 
System (AHSVS) 


Light Armoured Vehicle Remote 32 months as of June 2009 
Weapon Station (LAV RWS) (yet to be fielded) 


Source: Department of National Defence 


§.72 Recommendation. When National Defence plans urgent 
acquisitions, it should rigorously assess training requirements to ensure 
that there are a sufficient number of vehicles to meet training needs 
without reducing the number dedicated for operations. 


National Defence’s response. Agreed. In an effort to mitigate the 
risks associated with the aggressive processing of an unforecasted 
operational requirement, the Land Staff has embraced a revised strategy 
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across two broad fronts designed to more effectively address key project 
linkages such as training. First, abbreviated Capability Development 
assessments better define what is required to build, generate, employ, 
and sustain any given capability. With respect to training, the 
Directorate of Army Training, as well as Corps Schools/Centres of 
Excellence, play an increasing role in ascertaining what is required to 
train personnel for an emerging capability. On the second front, Army 
staffs closely shepherd training stocks once an unforecasted 
operational requirement capability is fielded. This typically involves a 
synchronized cycle whereby equipment and/or personnel are moved to 
facilitate the enhanced training of a unit or units for increased readiness 
in preparation for operations. 


The government approval process depends on good information 


5.73 When approving projects, the government makes decisions 
based on the information contained in documents it receives from 
government departments. The quality of this information is important 
for informed decision making, and departments are responsible for the 
content of these documents, including their accuracy. The Treasury 
Board of Canada Secretariat plays an oversight role in this decision- 
making process by performing a challenge and quality assurance 
function on draft documents and providing recommendations to 
government. This includes a review of the documents for consistency 
with government policy and department authorities. The Secretariat 
also provides feedback and advice to departments on draft documents. 


5.74 Our audit examined the challenge and quality assurance roles 
played by the Secretariat for the projects we audited. We interviewed 
its officials and reviewed files that the Secretariat provided to us. 


5.75 Officials at the Secretariat told us that the challenge function is 
largely an oral process with little written documentation. The quality 
assurance role is based mainly on the information presented in the 
documentation itself, and, therefore, in our view, it has its limits. 

The analysts at the Secretariat exercise due diligence by challenging 
the contents of draft documents based on compliance with Treasury: 
Board policy and department authorities. In doing so, they use their 
judgment and knowledge of the department and their perception of 
risk. They also help departments write clear and concise documents 
to government. 


5.76 Documentation we collected during the audit shows that the 
Secretariat reviewed and challenged submissions. We found that its 
officials met with department officials to discuss their draft government 
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documents and to be briefed on topics at hand. Questions were asked 
and clarification sought. For instance, we found that the Secretariat 
asked National Defence to provide justification for including an option 
to purchase 25 more vehicles in the RG-31 contract. The Secretariat 
also challenged National Defence on whether it fully understood the 
long-term cost implications of the tank replacement project. However, 
we found that with respect to these four projects, some important issues 
were not challenged by the Secretariat, and National Defence did not 
disclose all known relevant information. 


5.77 For two of these urgent projects, we examined whether the 
Secretariat had considered whether the projects could actually be 
managed in a manner that would respect the Treasury Board’s project 
management policies. We found that it had not. 


5.78 We also found that National Defence did not submit risk 
management strategies, known as the Project Profile and Risk 
Assessment (PPRA), to the Secretariat for two of the projects in our 
audit. This document is supposed to be provided to the Secretariat 
before draft submissions are prepared. Its purpose is to show that the 
Department has anticipated the possible risks related to the project 

_and has put in place mitigation strategies to deal with them. If the 
Secretariat had asked about the missing PPRAs, it would have learned 
that they had not been prepared for two of the projects. This would 
have given the Secretariat an indication that normal project 
management practices were not being followed. 


5.79 For the Light Armoured Vehicle Remote Weapon Station 

(LAV RWS) project, when National Defence informed the Secretariat 
that there would be delays and when it sought a second project approval 
for an additional $55 million, thus doubling the total cost of the vehicles 
to more than $100 million, the Secretariat did not challenge the 
Department as to whether this project remained the best option to 
deliver the needed capabilities and could still be considered urgent. 


National Defence did not disclose all relevant information 


5.80 Departments are expected to disclose all relevant information 
related to their submissions to government. Without this full 
disclosure, it is difficult for the Secretariat to fulfill its responsibilities to 
the government. We found that National Defence did not disclose all 
relevant information when it sought approval to lease 20 Leopard 2 
tanks. The Department stated that all its old Leopard | tanks being 
used in Afghanistan had to be replaced. We found that National 
Defence was aware, but did not disclose, that there was a significant _ 
risk that it would not be able to replace the entire Leopard 1 tank fleet 
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in Afghanistan because it would not be able to equip the new tanks 
with the land mine ploughs and bulldozer blades needed for operations 
there. National Defence’s project budget, as submitted to and 
approved by the government in spring 2007, included $2.4 million for 
ploughs and bulldozer blades. Subsequently, several of the Leopard | 
tanks had to remain in Afghanistan to provide this capability. 


5.81 The LAV RWS project is another case where, in our opinion, 
National Defence did not provide in its documents to the Secretariat 
and the government all the relevant information for informed decision 


making (Exhibit 5.6). 


Exhibit 5.6 National Defence gave inadequate information to the government during the LAV RWS 
project 


The purpose of the project was to acquire 33 vehicles called the Light Armoured 
Vehicle Remote Weapon Station (LAV RWS), a variant of the Department's existing 
fleet of Light Armoured Vehicles (LAV III).The project was to equip 33 LAV III chassis, 
already owned by National Defence, with a remote weapon system (RWS), instead of a 
turret, thereby reducing its weight by several thousand pounds and allowing for more 
armour protection. 


In October 2006, the government approved the purchase of 33 LAV RWS vehicles at a 
cost of $55 million. 


We found that the Department did not inform the government that it would later return 
for additional funding for custom armour kits. National Defence told us that the entire 
amount and scope of the project was not included in the first proposal because the 
details were not sufficiently developed at that time. In March 2007, the government 
approved $55 million more for custom armour kits and in April 2007 an additional 
$8 million for add-on armour. 


During the audit, we found that the reason National Defence decided to proceed 

with this project was different from what it told the government. National Defence told 
government that it needed to replace two vehicles: the inadequately protected 
G-Wagon and the Bison vehicle used for the administrative transport of troops. 
However, documents from National Defence senior management meetings consistently 
show that the project was approved in order to build a better LAV, starting with 
development of a prototype. 


National Defence also told the government that an options analysis showed that the 
LAV RWS was the best option to replace the G-Wagon and the Bison vehicles. 
However, we found that this was not the case. The Canadian Force’s options analysis 
for the Bison replacement recommended several options and the LAV RWS was not 
one of the preferred options. Furthermore, no analysis of the G-Wagon replacement had 
been prepared at that time. 


When National Defence requested an additional $55 million in the spring of 2007, 

it did not disclose the total cost, which would now be in excess of $100 million. !t also 
failed to inform the government of the status of the project in relation to the money 
approved in October 2006. 


At the time of this second government decision, little progress had been made on the 
project, which was supposed to be fully complete by February 2008. Now, it appears 
that the vehicles will be available in Afghanistan only sometime in 2009. National 
Defence informed us that the original project plans were unrealistic and did not 
adequately consider how complex the project was. 
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5.82 The Secretariat performs an important function in terms of 
ensuring that what is presented to government is accurate, complete, 
and supports the decisions that the government is being asked to 
make. We found that while the Secretariat has challenged the 
Department on some important issues, the challenge function has its 
limits. We found that for the projects we audited, the Secretariat did 
not question or challenge some important issues. The completeness 
and accuracy of submissions are the responsibility of the sponsoring 
Minister who presents them to the Treasury Board for consideration. 
We found instances where National Defence did not provide 
important information.to the government that was needed for 
informed decision making. 


5.83 Recommendation. National Defence should review its processes 
to ensure that all relevant information is included in its requests to the 
Treasury Board for project approvals. 


National Defence’s response. Agreed. As noted in this report, 
National Defence’s project approval process is consistent with 
Treasury Board direction and policy. That being said, the process is 
being reviewed to ensure that it can better accommodate the possible 
need to deal with urgent operational requirements. National Defence 
will be carefully examining lessons learned from the projects included 
in this report, in conjunction with an analysis of the current project 
approval guidance, to ensure that relevant information is not excluded 
when project approvals are being fast-tracked for urgent projects. 


Conclusion 


§.84 Our audit examined four urgent vehicle acquisitions for use in 
Afghanistan. We found that the three vehicles that have been 
deployed to Afghanistan, according to National Defence, have met 
operational needs. However, not all the projects have delivered all the 

, required capabilities and National Defence has had to make 
adjustments. 


5.85 Unlike non-urgent acquisitions, the Armoured Patrol Vehicle 
(RG-31), the Leopard 2 Tank replacement project, and the Armoured 
Heavy Support Vehicle System (AHSVS) project were procured and 
delivered quickly and, in the opinion of National Defence, have 
contributed to the safeguarding of Canadian soldiers in Afghanistan. 
However, the Light Armoured Vehicle Remote Weapon System 

(LAV RWS) project is nearly two years behind the original schedule 
and will cost at least double what the government was first told. 
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5.86 The two competitive processes that we audited were successful in 
contracting the needed vehicles in a timely manner and in complying 
with all rules that apply to contracting. However, we did find some 
problems with contracting roles and responsibilities, information given 
to senior officials, and contract administration. Given the urgent 
operational requirements of National Defence, the problems we saw in 
the competitive processes were somewhat understandable. 


5.87 National Defence should examine its Project Approval Guide to 
determine how acquisitions can be managed so that urgently needed 
equipment can be acquired in a timely manner while respecting 
accepted project management principles. We found that the 
documentation required for these urgent projects was not produced in 
a manner that complies with the Department’s own Guide. 


5.88 In order to use new equipment, there must be adequate training. 
In one project, the Department underestimated the need for vehicles 
that could be used for training. This means that the number of vehicles 
available for operations will be significantly reduced. 


§.89 While our audit found that three projects slipped behind 
schedule, we recognize that the timelines were ambitious. We found 
that some projects, such as the RG-31, which was in production, can 
be quickly delivered to where they are needed. Other projects, such as 
the LAV RWS that involve development work, require more time to 
plan. When this need for planning time is not respected, the project 
may fall short in meeting operational needs and also suffer cost 
overruns. In our opinion, given the significant development work in 
the LAV RWS project, it should have been managed more rigorously 
in accordance with National Defence’s Project Approval Guide. 


5.90 tis important that Treasury Board ministers have the information 
they need to make good decisions. While we found that the Treasury 
Board of Canada Secretariat exercised its challenge function, we also 
found that it missed some important issues related to these projects. 

As well, National Defence did not disclose all relevant information. 
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About the Audit 


All of the audit work in this chapter was conducted in accordance with the standards for assurance 
engagements set by The Canadian Institute of Chartered Accountants. While the Office adopts these 
standards as the minimum requirement for our audits, we also draw upon the standards and practices of 
other disciplines. 


Objectives 
The objectives of the audit were to determine whether 


« National Defence can demonstrate that the purchase of selected military vehicles used in Afghanistan 
met operational needs and were acquired in a timely manner; and . 


* Public Works and Government Services Canada and National Defence purchased these selected 
military vehicles in compliance with applicable contracting legislation, regulations, and policies and 
acquired them in a timely manner. | 


Scope and approach 


Our audit focused on four vehicle acquisitions undertaken by National Defence for use in Afghanistan. 
We selected these four projects based on the following: 


« Mission critical. The vehicles were urgently needed to help save soldiers’ lives and to carry out 
military operations in Afghanistan. 


- Cost. The cost of the project was at least $100 million. 


Acquisitions made by the Canadian Special Operations Forces Command have been excluded from our 
audit scope. As well, the audit did not examine Industrial and Regional Benefits. 


The majority of the audit work was conducted at the offices of National Defence located in the National 
Capital Region. This mostly involved file reviews and interviews with officials involved in the selected 
projects. We obtained information from many commands within National Defence, including the 
Assistant Deputy Minister of Materiel, Army, Canadian Operational Support Command, Canadian 
Expeditionary Force Command, and Vice Chief of the Defence Staff. 


The audit also included a review and analysis of various internal reports generated by Joint Task Force 
Afghanistan, including commanders’ reports, equipment status reports, logistic reports, and JED incident 
reports. 


We conducted procurement file reviews and interviews with staff at Public Works and Government 
Service Canada located in the National Capital Region and in London, Ontario. 


We also reviewed files held by the Treasury Board of Canada Secretariat in connection with Treasury 
Board submissions and interviewed current and former analysts. 


We conducted limited interviews with company officials. 
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Listed below are the criteria that were used to conduct this audit and their sources. 


Criteria — 


Sources — ‘ 


We expected that when acquiring selected military vehicles for 
Afghanistan, National Defence would have complied with its 
Project Approval Guide. 


We expected National Defence to have: 
¢ identified a capability deficiency 

¢ completed an options analysis 

e defined the project 

¢ implemented the purchase 

* completed a close-out 


We expected that National Defence could demonstrate that the 
projects delivered a solution to the problem outlined in the 
statement of capability deficiency. 


_ Timeliness was an implied criterion, given that these were 
urgent projects required for operations in Afghanistan. 


* Treasury Board of Canada, Project Approval Policy 
* Treasury Board of Canada, Project Management Policy 


¢ Department of National Defence, Project Approval Guide 


We expected that the Treasury Board of Canada Secretariat 
would have provided a challenge function by ensuring that 
submissions to the Treasury Board of Canada were thoroughly 
analyzed and properly prepared. 


¢ Treasury Board of Canada Secretariat, Guide to Preparing 
Treasury Board Submissions (2007) 


We expected that Public Works and Government Services 
Canada (PWGSC) and National Defence would have carried out 
the acquisitions in compliance with all laws, regulations, and 
policies that apply. 


e Treasury Board of Canada, Contracting Policy 
e Financial Administration Act, sections 32, 33, and 34 
¢ Government Contracts Regulations 


¢ Public Works and Government Services Canada, Supply 
Manual 


We expected that PWGSC and National Defence would have 
worked together to complete the acquisitions in a timely 
manner. 


¢ Public Works and Government Services Canada, Supply 
Manual 


Management reviewed and accepted the suitability of the criteria used in the audit. 


Period covered by the audit 


The audited activities occurred between June 2005 and May 2009 inclusively. 


Audit work for this chapter was substantially completed on 29 May 2009. 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 5. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


Project management 


5.22 National Defence, in 
consultation with the Treasury Board 
of Canada Secretariat, should 
examine its Project Approval Guide 
to determine how the Department 
can manage the process of acquiring 
urgently needed equipment in a 
timely manner while respecting sound 
project management principles and 

- Treasury Board policies. (5.14-5.21) 


Contracting 


§.40 For urgent operational 
requirements, National Defence should 
involve Public Works and Government 
Services Canada (PWGSC) early, in 
accordance with agreed upon roles and 
responsibilities, so that effective 
procurement strategies can be 


implemented to meet operational 
objectives. (5.23-5.39) 


Agreed. National Defence will review the current project 
approval guidance with a view to align the present 
documentation requirements for procurement greater than 

$1 million with the project approval framework as outlined in 
the Project Approval Guide. The Treasury Board of Canada 
Secretariat and other stakeholders will be consulted in order to 
achieve the correct balance between urgency and appropriate 
levels of documentation to support decision-making rationale. 
The National Defence group responsible for materiel will be 
consulted to ensure that coordination of the Guide and Project 
Management Manual meets the requirements. 


Agreed. Lessons learned from recent procurements, and 
opportunities for making process and other improvements, are 
being explored in some depth as part of a number of ongoing 
government reviews of defence procurement. Measures to 
further improve the already close collaboration between 
National Defence and PWGSC, while respecting the statutory 
responsibilities of the respective departments, are being 
examined as part of these reviews. 


In addition, PWGSC Acquisitions Branch now has full 
membership in several standing and ad hoc National Defence 
bodies that oversee the planning of future acquisitions, both 
routine and urgent. 
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5.48 Public Works and Government 
Services Canada (PWGSC) and 
National Defence should examine 
lessons learned in contracting for 
urgent operational requirements that 
could be applied to help speed up the 
regular procurement process. 


(5.41-5.47) 


Department of National Defence’s response. Agreed. National 
Defence continues to work with PWGSC and other key partners 
to develop a whole-of-government approach to improve and 
accelerate military procurement. Lessons learned from recent 
projects related to urgent operational requirements are being 
evaluated and contributing to these ongoing efforts. In 
particular, National Defence is working with PWGSC to develop 
a comprehensive defence procurement process model that will 
help optimize the process for both routine and urgent military 
acquisitions. Jo this end, National Defence has largely 


~ completed the initial mapping of its current internal processes 


and plans to have the full National Defence model completed by 
31 March 2010. 


PWGSC’s response. Agreed. The observations noted in the 
chapter regarding the implementation of strategies to fast track 
the procurement and delivery of the urgently required military 
vehicles, while respecting government contracting policies and 
objectives, reflect the success of these PWGSC military 
procurement initiatives. 


PWGSC will conduct a lessons-learned exercise to review the 
procurement activities undertaken during the urgent acquisition 
of the four types of military vehicles destined for Afghanistan, 
examined during this audit. The lessons learned will highlight 
the steps, activities, actions, and processes that led to the quick 
and timely acquisition of these vehicles. 


The lessons learned will be completed by the end of October 
2009 and, where applicable, effective practices identified 
through the lessons-learned exercises would be considered for 
implementation as part of the Department’s existing initiative 
aimed at improving the efficiency of major military and other 
Government of Canada procurements. 
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§.72. When National Defence plans 
urgent acquisitions, it should rigorously 
assess training requirements to ensure 
that there are a sufficient number of 
vehicles to meet training needs without 


reducing the number dedicated for 
operations. (5.49-5.71) 


ACQUIRING MILITARY VEHICLES FOR USE IN AFGHANISTAN 


Agreed. In an effort to mitigate the risks associated with the 
aggressive processing of an unforecasted operational 
requirement, the Land Staff has embraced a revised strategy 
across two broad fronts designed to more effectively address key 
project linkages such as training. First, abbreviated Capability 
Development assessments better define what is required to build, 
generate, employ, and sustain any given capability. With respect 
to training, the Directorate of Army Training, as well as Corps 
Schools/Centres of Excellence, play an increasing role in 
ascertaining what is required to train personnel for an emerging 
capability. On the second front, Army staffs closely shepherd 
training stocks once an unforecasted operational requirement 
capability is fielded. This typically involves a synchronized cycle 
whereby equipment and/or personnel are moved to facilitate the 
enhanced training of a unit or units for increased readiness in 
preparation for operations. 


Oversight by the Treasury Board of Canada Secretariat 


5.83 National Defence should review 
its processes to ensure that all 
relevant information is included in its 
requests to the Treasury Board for 
project approvals. (5.73—5.82) 


Agreed. As noted in this report, National Defence’s project 
approval process is consistent with Treasury Board direction and 
policy. That being said, the process is being reviewed to ensure 
that it can better accommodate the possible need to deal with 
urgent operational requirements. National Defence will be 
carefully examining lessons learned from the projects included in 
this report, in conjunction with an analysis of the current project 
approval guidance, to ensure that relevant information is not 
excluded when project approvals are being fast-tracked for 
urgent projects. 
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What we examined 


Why it’s important 


Land Management and 
Environmental Protection on Reserves 


Main Points 


Land management refers to managing the use and development of land 
resources in a sustainable way. Land can be used for a range of purposes 
that interact and may compete with one another, making it desirable 
to plan and manage land use in an integrated way that provides for 
protecting the environment from negative impacts of economic 
development. 


Until the early 1980s, all land management activities on First Nations 
reserves were the exclusive responsibility of the federal government. 
However, through comprehensive land claim settlements and 
self-government agreements, several First Nations have assumed full 
control and responsibility for the development of the reserve lands; 
some others have assumed responsibility under other arrangements 
for various aspects of land management on their reserves. The specific 
roles and responsibilities of the federal government vary depending 
on its agreement or arrangements with the First Nation. 


We examined how Indian and Northern Affairs Canada (INAC) 

and Environment Canada have carried out the federal government’s 
responsibilities for land management and environmental protection on 
reserves. This included looking at regulatory and non-regulatory 
measures used to manage the environment and the support INAC 
provides to those First Nations wishing to assume more land 
management responsibilities. 


Audit work for this chapter was substantially completed on 
29 May 2009. 


Reserve lands are central to First Nations peoples’ history, cultural 
identity, and day-to-day activities. Many First Nations are among 
the most economically deprived communities in the country. Their 
sustainable economic development depends on their access to and 
control over their land and natural resources and on a clean and 
healthy environment. Without the capacity and means to develop 
and use their lands and resources sustainably for their economic 
benefit, the opportunities for First Nations to improve their quality 
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What we found 
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of life and approach the standard of health and well-being enjoyed by 
other communities in Canada are severely restricted. 


« Indian and Northern Affairs Canada and Environment Canada have 
identified a significant gap between First Nations reserves and 
Canadian communities elsewhere in the degree to which regulations 
protect the environment. While the federal government has the 
authority to regulate environmental threats on reserves, it has rarely 
used this authority to develop regulations to mitigate environmental 
threats that are regulated off reserves by provincial governments. 


¢ INAC has done little to monitor and enforce compliance with 
existing regulations. For example, while regulations under the 
Indian Act require a permit issued by INAC to operate a landfill site 
or burn waste on reserve lands, the Department has issued few 
permits and is not equipped to conduct inspections, monitor 
compliance, and enforce the regulations. Consequently, garbage is 
often not confined to licensed landfill sites and there is no 
monitoring of the impacts on drinking water sources and air quality. 
Off reserves, provincial and municipal regulations and enforcement 
help to prevent such situations. 


« Although INAC has developed legislative and program options to 
support First Nations who wish to assume partial or full control of 
land management on their reserves, most First Nations lands are still 
managed by the Department under the Indian Act. First Nations’ 
access to alternative land management regimes established by INAC 
does not meet the demand. Consequently, INAC is unable to keep 
its commitment to transfer greater control of land management to 
First Nations who want it and are ready to take on these 
responsibilities. Furthermore, for First Nations under either of the 
alternative land management regimes, the Department provides 
insufficient training in comparison with the land management 
responsibilities it is transferring to them. 


« Officials of both INAC and Environment Canada cited a lack of 


funding as a key reason for not meeting some of their commitments. 


The departments have responded. The departments agree with all of 
our recommendations. Their detailed responses follow each 
recommendation throughout the chapter. 
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Introduction 


6.1 Land management refers to managing the use and development 
of land resources in a sustainable way. Land can be used for a range of 
purposes that interact and may compete with one another. This makes 
it desirable to plan and manage land use in an integrated way that 
provides for protecting the environment from the negative impact 

of economic development. Historically, Indian and Northern Affairs 
Canada (INAC) has been responsible for administering the land 
management requirements of the Indian Act in response to requests 
by First Nations. It is still responsible for all land management 
activities for the majority of First Nations. The Department spent over 
$135 million on managing reserve lands in the 2006-07 fiscal year. 


6.2 According to INAC, Canada has 615 First Nations, with more 
than 80 percent of them consisting of fewer than 1,000 people living 
on reserves. First Nations people have higher unemployment rates and 
lower average incomes than other Canadians, particularly those living 
on reserves. Many First Nations view the development of reserve lands 
as their most viable means of generating wealth for their communities. 
Currently, 568 First Nations have land spread over more than 

3,000 reserves, with a total area equivalent to about 60 percent that of 
Nova Scotia. Just over 160 reserves are urban; the remainder are rural 
and remote. 


6.3 Fifty-six percent of First Nations members across Canada live 
in reserve communities; with some exceptions, these communities 
include homes, churches, administration buildings, and schools built 
on reserve land. In addition, some First Nations use portions of their 
land for economic development—for example, by leasing it to other 
parties or developing natural resources. 


6.4 How these lands are managed, regulated, and protected is 
important to First Nations and to the well-being of their communities. 
With populations growing at a faster rate than Canada’s population, 
these reserve lands will need to accommodate increasingly more 
homes, facilities, and infrastructure. 


6.5 In communities off reserves, a series of municipal, provincial, 
and federal laws and regulations govern land use, occupancy, and 
development. However, for the most part, provincial and municipal 
laws and regulations related to land use do not apply to reserves. 
For most First Nations, the Chief and council can pass bylaws 

and develop community plans for their reserves, but the federal 
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government has the authority and responsibility to regulate use and to 
protect the environment of reserve lands. 


6.6 Since 1876, the Indian Act has been the main instrument 
governing the use of First Nations lands. For a number of years, First 
Nations and others have raised concerns about how reserve lands have 
been managed under the Act. 


Concerns about the management of reserve lands 


6.7 The Indian Act has shaped how First Nations have been able 

to develop and profit from their reserve lands. The 1996 Royal 
Commission Report on Aboriginal Peoples and the 2007 report by the 
Senate Standing Committee on Aboriginal Peoples, Sharing Canada’s 
Prosperity—A Hand up, not a Handout, both pointed to legislative 
and regulatory barriers and limited access to lands and resources as 
problems for First Nations’ economic development. The Senate 
Committee report identified the Indian Act as one of the critical 
barriers to economic development on reserves. It also concluded that, 
under the Indian Act, market forces do not operate properly on Indian 
lands, which substantially raises the costs of doing business there. 


6.8 Some of the Act’s provisions that affect land management date 
back more than a century. Many First Nations have identified the 
following provisions as making economic development today a 
significant challenge for them: 


¢ Control of moneys. Indian Act amendments in 1880 gave power 
to the federal government to decide how to spend revenue 
from the surrender and sale of reserve lands or other resources. 
First Nations must ask to use this revenue, which is held by the 
government in trust accounts, and specify how they intend to 
use it in each case. First Nations have noted that this inhibits 
those who wish to use these moneys to develop reserve lands. 


¢ Control of resources. The written permission of the Minister 
of Indian and Northern Affairs is needed to remove minerals, 
timber, hay, and other resources from reserves. Title to resources 
on reserves does not lie with the First Nation, but with the Crown, 
for the benefit and use of the First Nation. A First Nations band 
Chief and council are thus not at liberty to enter into resource 
extraction agreements with industry without the willing, active, 
and timely participation of INAC. 
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¢ Inability to use lands as collateral. The Indian Act includes 
provisions that prohibit the seizure of property on reserve lands 

~ by outside interests, including banks and other would-be 
investors. Therefore, reserve lands cannot be used as collateral. 
This has significantly limited First Nations’ ability to secure 
financing for investment in economic development activities on 
reserves. 


¢ Outdated regulatory environment. The land management 
provisions of the Indian Act have not been significantly amended 
since 1951; they do not address economic development or 
environmental protection issues that, off reserves, are regulated by 
a combination of federal, provincial, and municipal institutions. 


6.9 First Nations have also expressed concern that the processes that 
support land management under the Indian Act are cumbersome and 
slow. First Nations are governed by about 35 land-related sections of 
the Indian Act, including about 25 provisions that involve the Minister 
or Governor in Council in reserve land and resource management 
decisions. Depending on the type of land transaction, the process 
involves a First Nations band council resolution informing INAC of a 
proposed land use change, such as a new lease. A departmental lands 
officer works with the First Nation to complete the land transaction. 
Sometimes the First Nation’s membership has to vote in a referendum 
on specific proposals. The lands officer reviews the proposal, ensuring 
that the change is.consistent with any land use planning in place and 
that it conforms with pertinent environmental requirements. It must 
then be approved by the Minister and registered. First Nations and 
INAC officials have indicated that land management provisions of 
the Indian Act create impediments to the economic development of 
reserves because of the lack of timely decision making. 


6.10 As well as fostering the economic well-being and prosperity 

of First Nations, one of INAC’s primary objectives is to encourage 
sustainable management of First Nations lands and resources. INAC 
and Environment Canada both have responsibilities for protecting 
the environment on reserve lands, and the federal government is 
committed to providing the same standards of environmental 
protection that other Canadian communities have come to expect. 
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Transfer of control of land 
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management 


Focus of the audit 


6.11 This audit focused on whether INAC and Environment Canada 
have met their responsibilities for land management on reserves, 
specifically with respect to transferring control of land management 
and providing essential environmental protection on reserve lands. 
We assessed whether INAC had developed legislation and regulations, 
programs, or policies that support First Nations’ land management as 
alternatives to managing under the Indian Act. We also examined 
whether INAC and Environment Canada had provided essential 
protection of the environment on First Nations reserves, by developing 
and implementing legislative options, programs, and policies, and 
whether environmental protection measures on reserves were 
comparable with those off reserves. 


6.12 We examined the legislative options, institutions, and programs 
developed since 2005, paying particular attention to the actions of 
departments over the past three years. We did not audit land activities 
related to 41 First Nations whose lands are managed through 
comprehensive land claims and self-government agreements. 


6.13 More details on the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


6.14 Given the concerns raised by First Nations, the 1996 Royal 
Commission on Aboriginal Peoples, the Senate Standing Committee 
on Aboriginal Peoples, and the Department’s own assessments, we 
expected that Indian and Northern Affairs Canada (INAC) would 
develop alternative ways of dealing with land management issues to 
provide First Nations with greater control to address some of the ° 
economic development challenges particular to reserve lands. 
Specifically, we expected that, consistent with commitments in its 
reports on plans and priorities, INAC would have proposed legislative 
or regulatory measures, developed institutions, and put in place 
programs or other arrangements that would support management of 
reserve lands. 


6.15 We reviewed legislative and program changes made over the past 
five years, obtained and assessed related documentation, interviewed 
department officials, and spoke with a number of First Nations 
representatives. 
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The Department developed legislative and program options to support First Nations’ 
management of their lands 


6.16 We found that the Department had worked with First Nations to 
develop and propose four legislative measures, which were subsequently 
enacted, and to develop four related financial institutions (Exhibit 6.1). 
All of these initiatives were aimed at providing First Nations with 
greater flexibility in managing land management activities and 
responding to concerns related to land management on reserves. 


Exhibit 6.1 Recent legislative changes to provide for more flexible land management 


Indian Oil and Gas Act (revised), May 2009 


¢ Purpose: To update provisions that remained unchanged since 1974. 


. Key changes: Allows regulation-making powers, an enforcement system 
comprised of fines and penalties, and environmental protection. 


¢ Intent: To facilitate an oil and gas regime on reserves equivalent to those regimes 
off reserves. 


First Nations Oil and Gas and Moneys Management Act, November 2005 


¢ Purpose: To allow for First Nations control over oil and gas activities taking place 
on reserve lands and for the management of Indian moneys. 


¢ Key changes: Allows oil and gas development and/or moneys management 
transfers to the First Nation. 


e Intent: To provide for greater First Nations control over oil and gas development. 
First Nations Commercial and Industrial Development Act, November 2005 


¢ Purpose: To facilitate a regulatory regime for an on-reserve project that would 
mirror the regulatory regime off reserves. 


e Key changes: Allows federal regulations to be developed for individual projects 
on reserves that would be equivalent to those off reserves. 


¢ Intent: To remove any regulatory gaps between on-reserve and off-reserve 
development. 
First Nations Fiscal and Statistical Management Act, March 2005 


* Purpose: To allow First Nations to derive revenue from the collection of taxes 
from band lands. Assistance with regard to taxation, finances, and statistics 
collection will be carried out through four institutions. 


* Key changes: Allows First Nations to create laws for the taxation of reserve land 
and to access institutions mandated to develop capacity. 


¢ Institutions created: 


¢ First Nations Tax Commission assists in the development of property tax 
regimes on reserve. 


¢ First Nations Finance Authority (FNFA) issues bonds and raises long-term 
private capital. 

e First Nations Financial Management Board assesses, assists, and develops 
standards to allow First Nations to borrow from the FNFA. 


¢ First Nations Statistical Institute is mandated to provide statistical information 
on First Nations and build their statistical capacity. 
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6.17. These federal acts and institutions may enable First Nations to 
make more productive use of reserve land. To the extent that they are 
applied, these mechanisms could provide First Nations with access to 
capital and encourage more economic development activity on 
reserve lands. 


6.18 However, these acts and institutions were implemented only over 
the last few years. Some of them are voluntary and are accessed by First 
Nations at their own pace, and therefore they have affected only a 
small number of First Nations to date. We did not audit their 
implementation or assess their efficacy. We note that while these 
options are not designed to address all of the concerns expressed, the 
Department has developed mechanisms to address significant issues. 


6.19 In addition to the more recent legislative amendments and 
institutions that INAC has developed with First Nations, we examined 
various land management regimes developed by the Department. 
According to its Land Management Manual, INAC views the transfer 
of land management responsibilities to First Nations as part of an 
overall departmental approach to facilitating First Nations’ control 
over their communities. It has committed to moving forward, in 
partnership with First Nations, from a relationship characterized by 
dependency to one in which First Nations assume greater jurisdiction, 
authority, and control over their lands, resources, and environment. 
We therefore expected to find options that would facilitate this transfer 
of control over land management to First Nations. 


6.20 INAC has provided options since the early 1980s for First 
Nations who considered that the Indian Act regime of land 
management was not meeting their needs. The Regional Lands 
Administration Program (RLAP) was a co-management approach 
between INAC land managers and First Nations under the Indian Act. 
The Delegated Lands Management Program authorized First Nations 
to manage and deliver, on INAC’s behalf, specific land management 
services required under the Indian Act. Both programs still operate, 
with 95 First Nations participating, but have been closed to any 
additional First Nations since 2004. Instead, the Department has 
developed two other options to enable First Nations to assume more 
land management responsibilities on their reserves. | 


6.21. The Department collaborated with First Nations to establish a 
Framework Agreement on First Nation Land Management in 1996, 
which led to the passing of the First Nations Land Management Act 
(FNLMA) in 1999. In 2005, the Department put in place the Reserve 
Land and Environment Management Program (RLEMP), a pilot 
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program intended to replace both the RLAP and the Delegated Lands 
Management Program. We examined both of these initiatives. 


Land management alternatives do not provide enough access to First Nations who 
otherwise could participate 


6.22 Reserve Land and Environment Management Program 
(RLEMP). The RLEMP is a pilot program that transfers from INAC to 
First Nations certain responsibilities under the Indian Act, such as 
community land use planning, environmental management, natural 
resources management, compliance monitoring, and administration of 
land transactions. 


6.23 This program is open to First Nations that are currently 
operating under the RLAP or the Delegated Lands Management 
Program. To be eligible, a First Nation must have a land manager, at 
least one active registered land transaction, and a track record of good 
financial management. 


6.24. Through training and delegation of authorities, this program is 
intended to build First Nations’ land management capacity. It is also 
intended to accelerate the pace at which First Nations exercise greater 
control over their reserve land, resources, and environment, rather 
than having lands managed by the Department. 


6.25 The program began as a pilot program in 2005 with 15 First 
Nations. According to information that the Department published at 
the time, it expected the program to be available to all First Nations by 
April 2006. We therefore expected that INAC would have expanded 


the program to include any First Nations who wanted to participate. 


6.26 We found that access to the program was still limited to First 
Nations who were participating in either the RLAP or the Delegated 
Lands Management Program. Moreover, although some First Nations 
in these programs had indicated their interest in the RLEMP the 
Department has not always admitted them. The process of entering © 
the RLEMP is initiated by INAC when it invites a First Nation to send 
its land manager for RLEMP training. About 40 percent of First 
Nations under the two entry programs have moved into the RLEMP. 


6.27 INAC has not provided access to the RLEMP for the majority of 
First Nations. Thus it is not giving many First Nations the 
opportunities that the program offers to build their land management 
capacity and expertise in order to better manage economic 
development and environmental protection on their reserves. 
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Land code—A code that sets out the basic 
provisions governing the exercise of a First 
Nation's rights and powers over its reserve 
lands. It can include procedures for mortgage 
provisions, seizure of leasehold interests, 
dispute resolution mechanisms, and making 
and publishing First Nation laws. 
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6.28 First Nations Land Management Act. The First Nations Land 
Management Act (FNLMA) was developed to provide First Nations 
with greater flexibility in the areas of land, resource, and 
environmental management. It stems from the 1996 Framework 
Agreement on First Nation Land Management between 14 First 
Nations and the Minister of-Indian Affairs and Northern 
Development. The FNLMA effectively removes First Nations who 
choose this option from the land management provisions of the Indian 
Act. It enables them to adopt a land code and assume management 
over their reserve lands and associated revenues. It also provides First 
Nations with the law-making authority over environmental protection. 
Upon becoming operational under the FNLMA, a First Nation may 
not return to being governed under the land management provisions of 


the Indian Act. 


6.29 The FNLMA is open to First Nations who are operating under 
any of the existing land management regimes. The Framework 
Agreement on First Nation Land Management established a First 
Nations Lands Advisory Board consisting of representatives from First 
Nations operating within the regime. Once an interested First Nation 
applies, both INAC and the Board assess the First Nation’s 
preparedness to enter into this regime. The Department’s regional 
office considers the First Nation’s financial management status, any 
litigation involving reserve lands, the band’s election cycle status, 

and any environmental concerns and land interest issues. 


6.30 First Nations under the FNLMA may give their own consent for 
land transactions. This provision of the Act aims to enable First 
Nations to make timely business and administrative decisions and to 
accelerate their progress in economic development, resource 
management, and land use planning. Once a First Nation is fully 
operational in all provisions of the FNLMA, INAC’s role in managing 
the First Nation’s land activities is limited to maintaining the agreed- 
upon funding and registering land transactions as well as providing 
some technical support. 


6.31 For most First Nations, the FNLMA regime is, in effect, the only 
alternative to land management under the Indian Act. Thus, in light of 
INAC’s commitment to work with interested First Nations to enable 


_ them to assume control over their lands, resources, and environment, 


we expected that the Department would have accepted First Nations 
who were interested and ready into the FNLMA regime. 


6.32 When developing the programming for the FNLMA, the 
Department anticipated that about 90 First Nations could become 
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fully operational in all provisions of the Act. In 2002, its plans called 
for having 30 First Nations in the developmental phase of the FNLMA 
regime at any given time. The Department anticipated that up to 

75 First Nations would be operational by the end of 2007. At the time 
of our audit, only 22 First Nations were operational, 19 were in the 
developmental stage, 2 had negotiated self-government agreements, 
and 9 were inactive. Another 61 First Nations were waiting to enter 
the FNLMA, most of whom were still waiting to be assessed by the 
Department to determine whether they were ready to enter the 
FNLMA regime. At the end of 2008, 15 of these 61 First Nations 


had been waiting for more than seven years. 


6.33 The wait to enter the FNLMA regime could impede economic 
development for First Nations who want to obtain control over their 
land management. Carrying out complicated commercial transactions is 
difficult for First Nations under the Indian Act, due to the administrative 
delays caused by the required approvals and procedures. In addition, 
several sources indicate that private sector investors appreciate the 
certainty that comes from dealing directly with First Nations under the 
FNLMA, knowing that the First Nation, not INAC, holds the legal 
authority to make decisions. Until the Department can make the 
FNLMA or RLEMP accessible to First Nations who want to enter those 
regimes, it does not have the means for meeting its commitment to 
transfer greater control of land management to First Nations. 


6.34 Recommendation. Indian and Northern Affairs Canada should 
ensure that First Nations who are ready and who want greater 
autonomy over their land management can access either the Reserve 
Land and Environment Management Program or the First Nations 
Land Management Act regime. 


The Department’s response. Agreed. The Reserve Land and 
Environment Management Program (RLEMP) began as a pilot 
program in 2005 and has been funded internally since its inception. 
Existing authorities were used to allow for a pilot of the RLEMP 
limited to First Nations participating in existing programs. 


Since 1996, a number of First Nations have sought access to the First 
Nations Land Management Act (FNLMA) regime. In 2002, an 
amendment to the Framework Agreement on First Nation Land 
Management was made to allow up to 30 First Nations in the 
developmental phase of the FNLMA regime at any given time. 


In March 2008, INAC had to close the FNLMA regime to new 


entrants due to a lack of funding. The Department is currently seeking 
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additional resources to effectively implement the FNLMA regime for 
participants and expand it to new First Nations. 


In summer 2009, a sustainable funding source for the RLEMP was 

secured through the Aboriginal Economic Development Action Plan. 
With dedicated funding over four years, INAC will now be able to fully 
implement this program and allow up to 20 new First Nations per year 


to enter the RLEMP. 


Current options for delegated land management have not included sufficient 
training and capacity building 


6.35 When First Nations enter into a land management regime that 
provides them with greater responsibility for processing land 
transactions, they take over some of the land management activities 
that would otherwise have to be carried out by INAC. To carry out 
these responsibilities effectively, appropriate training for First Nations 
staff is essential. 


6.36 One of the priorities in the Department’s program activity 
architecture.is to build First Nations’ capacity for land management. 
The Department has also made specific commitments to facilitate 
training for First Nations in the RLEMP although not for First Nations 
in the FNLMA regime. While the First Nations Land Advisory Board 
has responsibilities to develop curricula and training under the 
FNLMA, we expected that INAC would have met its broad 
commitment to develop the capacity of First Nations taking on 
increased responsibilities, by ensuring that they are provided with 
appropriate training. We also expected that the Department would 
have met its specific commitments to provide training for First Nations 


under the RLEMP 


6.37 We looked at the training available to First Nations participating 
in the RLEMP and the FNLMA regime. We examined documentation 
on training curricula, training completed, and training availability. We 
interviewed officials of the Department, a First Nation organization, 
and a facility that provides RLEMP training. 


6.38 Reserve Land and Environment Management Program 
(RLEMP). One of the objectives of the RLEMP is to prepare First 
Nations for their roles and responsibilities through a training and 
certification program. First Nations land managers become certified 
when they successfully complete the RLEMP training program. This 
training program takes place over two years, with the land managers 
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participating in one- and two-week intensive classroom training 
sessions, followed by several weeks of home study to be completed 
while working in their communities. 


6.39 As soon as a First Nation member is registered for training, the 
First Nation is considered to be in the training and development phase 
of the RLEMP The First Nation becomes fully operational when its 
land manager completes a two-year training program. We found that of 
the 67 First Nations in the program at the end of our audit, only 28 
had certified land managers; the land managers in the other 39 were 
receiving training. This means that more than half of the First Nations 
with RLEMP responsibilities do not yet have fully trained land 
managers. 


6.40 While INAC covers all costs associated with the two-year 
RLEMP training program and certification, it plans to pay training 
costs for only one person per First Nation, with additional spots 
available for First Nations who want to cover the cost of training 
themselves. However, First Nations have expressed concern that 
training only one person per First Nation could have serious 
consequences. Staff turnover or unavailability of trained land 
managers for other reasons could result in First Nations being unable 
to carry out their land management responsibilities. 


6.41 We also found that training was sometimes arranged on short 
notice. Io commit to the first year of the training, the land manager 
must participate in three two-week sessions away from the community 
and another 28 weeks of home study while working within the 
community. For example, for first-year training scheduled to begin 

in April 2008, it was not until that month that the Department 
confirmed enrolment decisions and funding to the participants and the 
training facility. Late approvals mean participants have very little time 
to make arrangements for leaving their work and home responsibilities 
throughout the coming year. As a result, some First Nations staff were 
unable to take the training. Of the 30 participants offered entry to the 
program annually by the Department, only 23 started the training 

in 2008 and 16 in 2009. We observed that an INAC evaluation of the 
RLEMP pilot project in 2007 noted delays in approvals and funding. 


6.42 First Nations Land Management Act (FNLMA) regime. First 
Nations under the FNLMA are responsible for developing a land code, 
processing all their land transactions, working with INAC and 
Environment Canada to establish an Environmental Management 


~ Agreement, and passing their own environmental laws and regulations 
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related to land management. These responsibilities are more extensive 


than those assumed by First Nations under the RLEMP 


6.43 The development of training was to be addressed through an 
annual funding arrangement with the First Nations Land Advisory 
Board (FNLAB). INAC has.a funding arrangement that provides them 
about $3.5 million annually to, among other things, develop model 
land codes, laws, and management systems and to assist other First 
Nations seeking to become operational within the FNLMA regime. 


6.44 An additional element in this funding arrangement is the 
development of a training strategy. In the past, INAC has provided 
funding to the FNLAB specifically to develop curricula and training 
programs. In 2005, the Department discontinued this funding. Since 
then, the Department has offered priority for RLEMP training to 
selected First Nations under the FNLMA. 


6.45 At present, land management training is not mandatory for First 
Nations under the FNLMA. Of the 52 First Nations in this regime, 

10 had a land manager who had either registered for RLEMP training 
or was already certified at the time of our audit. Another 10 First 
Nations had started training but not completed it. 


6.46 The training developed for the RLEMP does not address all the 
responsibilities specific to the FNLMA. Without adequate training, 
those First Nations under the FNLMA may be unable to develop the 
necessary capacity but will still have full responsibility and liability for 
land management and environmental protection. 


6.47 Recommendation. Indian and Northern Affairs Canada, in 
collaboration with First Nations, should seek to ensure that an 
appropriate training program is developed for First Nations in the First 
Nations Land Management Act regime. The Department should also 
ensure that appropriate training is made available to First Nations in 
either the Reserve Land and Environment Management Program or 
the FNLMA regime. This training should address the requirements 
under each of the respective regimes. | 


The Department’s response. Agreed. The Reserve Land and 
Environment Management Program (RLEMP) has provided the only 
professional land management training available to First Nations for 
the past four years. Program evaluations and reviews of the RLEMP 
training program have concluded that the design and delivery of the 
training program are sound, credible, relevant and responsive to the 
needs of First Nations land managers and their communities. 
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From 1999 to 2005, Indian and Northern Affairs Canada (INAC) 
provided funding to the First Nations Land Management Resource 
Centre to develop a training strategy and curricula that would address 
the specific capacity requirements of First Nations Land Management 
First Nations. In 2005, INAC discontinued this funding due to a lack 
of progress. While the RLEMP training was designed for First Nations 
managing lands under the Indian Act, the Department decided to offer 
that training on a priority basis to First Nations Land Management 
First Nations as a temporary measure. 


A renewed effort by the Resource Centre to develop training and 
curricula specific to First Nations Land Management is currently being 
undertaken and INAC anticipates receiving a proposal to that effect 


in 2009-2010. | 


The RLEMP pilot project has been funded internally since its 
inception in 2005. The necessary sources of funds have not always 
been identified or available at the beginning of each fiscal year. As a 
result, there has been a delay in confirming whether the RLEMP 
training program could be offered in a given year and in notifying 
students of their acceptance into the training program. 


As mentioned previously, a sustainable funding source for the RLEMP 
was secured in spring 2009 through the Aboriginal Economic 
Development Action Plan. With dedicated funding for this program 
over four years, INAC will now be able to engage in the appropriate 
planning and forecasting required to notify new students slated to 
participate in the RLEMP training program much sooner than before. 


Fewer regulations protect the environment on reserves than off reserves 


6.48 An important part of land management is protecting the 
environment from adverse effects of economic and residential 
development and mitigating any risks to human health and safety. 
Indian and Northern Affairs Canada and Environment Canada have 
both expressed their commitment to promoting sustainable 
development on reserve lands. The use and development of land off 
reserves is generally subject to a wide range of provincial and 
municipal laws, regulations, and zoning codes, many of which are 
aimed at protecting the environment. 


6.49 In Canada, provincial, territorial, and municipal government 
laws and regulations provide measures to control a vast array of 
potential environmental threats. However, provincial and municipal 
laws and regulations generally do not apply on reserves and cannot be 
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enforced there. The federal government is responsible for all regulation 
of environmental protection on First Nations reserves that are 
managed under the Indian Act. It has power and authority under the 
Indian Act, the Canadian Environmental Assessment Act, the Canadian 
Environmental Protection Act, 1999, and the Fisheries Act to protect the 
reserve lands and foster their sustainable use. It also has other 
management tools, such as policies and programs, to promote 
environmental protection. 


6.50 In 1996, the parties who signed the Framework Agreement on 
First Nation Land Management identified four significant 
environmental threats: solid waste, sewage, fuel storage tanks, and 
environmental emergencies. In a 2007 study, Environment Canada 
also identified significant risks on reserves that required priority 
attention, including landfill, solid waste, and hazardous waste; 
discharge to surface water; and air emissions from incineration and 
open burning of garbage. 


6.51 Given the importance of these environmental threats to First 
Nations, and Environment Canada’s recognition of environmental 
risks, we expected that INAC and Environment Canada would have 
given priority attention to these matters and would have developed, 
monitored, and enforced regulations and taken other measures to 
address each issue. 


6.52 We compared the extent to which regulations applicable to 
reserves in the four priority areas identified by the parties were 
consistent with regulations that exist off reserves. Specifically, we 
compared regulations from selected legislation in Ontario with those 
from key legislation in place that address the same risks on reserves. 
We found that many of these priority environmental threats are not 
regulated on reserves (Exhibit 6.2). 


6.53 In several studies and evaluations since 2005, Environment 
Canada has confirmed the existence of a gap in environmental 
regulations on reserves. INAC has also recognized the existence of a 
regulatory gap. The First Nations Commercial and Industrial Development 
Act includes the power to pass regulations that mirror those of the 
province in which the reserve is located; however, to date, it has only 
been used to develop regulations for one industry sector on one reserve. 
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Exhibit 6.2 Regulatory provisions of selected environmental threats—comparison of regulations in place in the province of Ontario and 
on First Nations reserves 


Solid waste 


Waste disposal 


¢ waste can only be deposited in an approved waste disposal site ie) & 
Landfill sites 

¢ permit or certificate required to establish and operate S 2 

¢ standards for locating, operating, and maintaining landfill sites & @° 

* cannot use land previously used for landfill for 25 years (unless approved by the Minister) 6 C) 
Waste processing or recycling 

* permit required to establish and operate é @ 
Hazardous waste ~ 

* certain Sas designated as hazardous wastes & a] 

* hazardous waste tracked from cradle to grave (from generator to carrier and to receiver) @ © 

* restrictions on land disposal of untreated hazardous wastes & C) 
Sewage and other wastewater discharges 

Discharges/releases of pollutants into water 

* general prohibition on releases into water ; & & 
Septic systems 

Small systems (e.g. to serve a residence) 

* minimum standards for design and installation; permits and inspections required & @ 

Large and communal systems | . 

* can only be constructed and sete if covered by an approval/permit é C) 
Municipal/community wastewater treatment facilities 

* can only be operated if covered by an approval/permit & eo. 

* discharges must be in accordance with permit requirements (discharge limits, etc.) § © 

¢ facility operators must be licensed C] ©) 
Industrial/commercial facilities that discharge into water 

* can only be operated if covered by an approval/permit Fs) eo 

¢ discharges must be in accordance with permit requirements (discharge limits, etc.) & O? 
© discharge standards for persistent toxics for certain industrial sectors r @° 
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Exhibit 6.2 Regulatory provisions of selected environmental threat-—comparison of regulations in place in the province of Ontario and 
on First Nations reserves (continued) 


Underground and above-ground storage tanks 


e° 


* design and performance standards and requirements related to installation, leak detection, 
upgrades, registration, operation, and maintenance 


Propane facilities 


* National Propane Storage and Handling Code adopted 


Spills of pollutants and spill response (general) 


In the event of a spill of a pollutant, persons involved in spill must 
¢ report the spill to environmental authorities, the local municipality, and others @’ e 
* clean up the spill and restore the natural environment i @’ @e® 


Designated “dangerous goods”: accidental releases during transport 


¢ immediate reporting to provincial authorities and others & eB 


& environmental risk addressed through regulations 


©) environmental risk not addressed through regulations 


Explanatory notes 

Risk is addressed through permits (see paragraph 6.58). 

Conditions may be imposed under funding agreements and the Canadian Environmental Assessment Act. 

Fisheries Act regulates discharges from some industry sectors. 

Regulations apply to nine industry sectors. . 

Regulations apply to two industry sectors. 

Regulations do not apply to above-ground tank systems for heating or emergency power (2500L or less overall capacity). 
Regulations are for pollutants generally. 


Baer ae an a 


Regulations are for specified hazardous or flammable substances. 


Source: Ontario laws and regulations and federal laws and regulations 
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Existing regulations and other measures do not protect First Nations from priority 
environmental threats 


6.54 In 2002, INAC developed an Environmental Stewardship 
Strategy to protect the health and safety of First Nations and Inuit 
peoples, address the environmental regulatory gap, and reduce the risk 
of contamination as well as the Department’s potential financial 
liability. This strategy included both short- and long- term action items 
to develop and implement policies, directives, operational guidelines, 
standards, and other management tools. Under the strategy, the 
Department identified commitments and goals related to waste 
management, contaminated sites, and fuel storage tank management. 
In 2007, the Department developed a framework for implementing the 
strategy and monitoring progress. This framework included 
performance objectives, indicators, and targets. 


6.55 We therefore expected that the Department, in conjunction with 
Environment Canada, would have taken steps to fill the regulatory gap 
in order to protect reserve lands from environmental risks associated 
with land use. We also expected that INAC would have made progress 
toward its identified objectives. We reviewed INAC’s assessment of its 
progress. | 


6.56 We examined existing regulations and measures in place as well 
as actions taken by INAC, under its Environmental Stewardship 
Strategy, and by Environment Canada to address environmental 
threats identified as priorities to First Nations. We also reviewed the 
environmental regulations that currently apply on reserve lands for 
these priority areas. We looked at how the regulations are applied 
along with other measures taken to manage risks on First Nations 
reserves in British Columbia, Saskatchewan, and Ontario, the 
provinces with the greatest number of registered land transactions over 
the past three years. 


6.57 We found that the Department had implemented most of the. 
short-term action items in its Environmental Stewardship Strategy, 
such as developing an overall policy to guide the integration of 
environmental management practices. However, the Department’s 
performance report of its framework for implementing the Strategy 
identified many other action items that had not been fully carried out 
or that had been delayed. For example, the Department had planned 
to develop a policy specific to waste management by March 2008 and 
to develop service standards for waste management thereafter. By the 
end of our audit, the Department had developed only a draft policy 
and had not developed any standards. 


Chapter 6 | 19 


LAND MANAGEMENT AND ENVIRONMENTAL PROTECTION ON RESERVES 


20 Chapter 6 


6.58 Solid waste management. Under the 1978 Indian Reserve Waste 
Disposal Regulations, no person shall operate a garbage dump or use any 
land to dispose of, burn, or store waste without a permit that specifies 
terms and conditions for operating a landfill as well as environmental 
protection requirements. Permit holders are subject to an inspection 
before a permit will be renewed. INAC is responsible for issuing 
permits and enforcing these regulations. 


6.59 According to information provided by INAC, there are about 
365 First Nations in British Columbia, Saskatchewan, and Ontario 
whose land is managed by the Indian Act. About 150 of these First 
Nations have service agreements with municipalities to manage their 
solid waste. We expected that INAC would have issued at least one 
landfill permit for each of the remaining First Nations. We found that 
only 14 permits had been issued (all in Saskatchewan) under the Indian 
Reserve Waste Disposal Regulations. This means that about 200 First 
Nations in these three provinces should have had permits but did not. 


6.60 The Department has provided funding for the construction, 
operation, and maintenance of waste management facilities in about 
120 First Nations. It therefore had some information on waste 
management facilities in these communities but had no information on 
how solid wastes are disposed of in about 80 First Nations. 


6.61 Although INAC has the information to help it identify which 
First Nations do not have municipal service agreements or waste 
disposal permits, we found that the Department neither promotes nor 
conducts significant surveillance on reserves to look for illegal dump 
sites or garbage burning and is not equipped to monitor compliance, 
conduct inspections, or enforce the regulations. 


6.62 INAC officials explained that they do not issue permits or 
conduct surveillance, in part because the regulations are out-of-date 
and do not reflect the complexity of modern waste management 
systems. In addition, the Department considers penalties for 
non-compliance to be ineffectual (dumping or burning garbage 
without a permit in a residential community can result in a fine of not 
more than $100 or a three-month jail sentence, or both). In contrast, if 
the same activities jeopardize fish habitat, they can bring fines of up to 
$300,000 and/or a prison term of a maximum of six months under the 


Fisheries Act. 


6.63 Aside from requirements in the 14 landfill permits issued to 
First Nations, INAC has not developed any minimum regulatory or 
other standards for locating, operating, and maintaining landfill sites. 
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For example, such standards could include minimum distance from a 
body of water or a residential community. Provincial regulations 
include such standards, which apply off reserves. 


6.64 Sewage and other wastewater discharges. Environment 
Canada identified the discharge of household sewage and industrial 
and commercial wastewater into surface water as a significant risk on 
reserves. Like solid waste, liquid waste falls under the Indian Reserve 
Waste Disposal Regulations. Nevertheless, this is an area where the 
regulatory gap on reserves is particularly pronounced compared with 
the province of Ontario. Legislation and regulations in Ontario control 
sewage and other wastewater entry into groundwater and surface 
waters through approvals, permits, and other requirements. They 
cover, among other things, household septic systems and municipal 
sewage treatment plants. On reserves, there are no comparable 
regulations or standards in place. Moreover, we found that INAC does 
not issue permits for sewage treatment and disposal and makes no 
other effort to apply the existing regulations. 


6.65 INAC and Health Canada have taken some actions to address 
sewage treatment and disposal. These departments, working with First 


~ Nations, developed the national First Nations Water Management 


Strategy. After a national assessment of the water and wastewater 
systems on reserves in May 2003, INAC announced the First Nations 
Water and Wastewater Action Plan in 2008. It also launched an 
initiative in 2009 to develop a legislative framework for drinking water 
and wastewater in First Nations communities. Public consultation on 
this initiative began in February 2009. 


6.66 The Fisheries Act includes general provisions that apply 
throughout Canada and prohibits the release of deleterious substances, 
including sewage from reserves, into waters frequented by fish. 
Environment Canada is currently developing new Canada-wide 
regulations, under the Fisheries Act, to mitigate contaminants from 
community wastewater treatment facilities. These regulations are 
being developed in conjunction with provincial and territorial 
governments, with a view to phasing in regulations for all wastewater 
systems across the country over the next 30 years. Environment 
Canada expects that these regulations will be published in the Canada 
Gazette by the end of 2010. Environment Canada has indicated that 
wastewater systems that pose the greatest risk to the environment 
would have to comply with the new regulatory standards 10 years after 
the regulations come into effect. 
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6.67 Fuel storage tanks. In 2008, the government promulgated the 
Storage Tank Systems for Petroleum Products and Allied Petroleum Products 
Regulations under the Canadian Environmental Protection Act, 1999. 
These regulations, scheduled to be fully implemented by June 2012, 
establish standards for fuel storage tanks and require contingency plans 
to be in place for environmental emergencies. Under the regulations, 
fuel storage tanks must be registered by June 2010. Environment 
Canada, with INAC’s participation, provides information on these 
regulations on its website and in workshops delivered on reserves. 


6.68 In anticipation of these regulations, in 2006 INAC began 
carrying out an assessment of fuel storage tanks on reserves using funds 
for assessments of contaminated sites. As of February 2009, it had 
completed an inventory of 90 percent of fuel storage tanks in British 
Columbia and 30 percent of tanks in Ontario. In Saskatchewan, the 
Department had carried out an inventory of 191 fuel storage tanks, 
focusing only on tanks over 4,000 litres. Later that year, the 
Department concluded that storage tanks on reserves are not the 
property of the federal government and therefore the regulations did 
not require that INAC complete an inventory and register fuel storage 
tanks on reserves. The Department continued to carry out an 
inventory of fuel storage tanks in British Columbia and Ontario but 
stopped collecting information on fuel storage tanks in Saskatchewan. 


6.69 Environmental emergencies. Another area considered essential 
for environmental protection is the need to address environmental 
emergencies. Several provinces, including Ontario, have legal 
provisions to address environmental emergencies such as spills. For 
example, in the event of a pesticide spill, a spill report and cleanup 


would be required. However, such provincial requirements, which 


apply to spills of pollutants generally, do not apply on reserves. 


6.70 Under the Environmental Emergency Regulations of the Canadian 
Environmental Protection Act, 1999, reporting and other remedial 
actions are required in the event of an accidental release of specified 
harmful substances. These provisions apply throughout Canada, 
including on reserve lands. 


6.71 Asan additional means of mitigating risk, INAC has signed 
agreements with some provinces to coordinate planned actions in the 
event of environmental emergencies. 


6.72 Closing the gap. The environmental regulatory gap is 
particularly significant for First Nations that enter into the First 
Nations Land Management Act regime. In entering into this regime, a 
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First Nation becomes responsible for closing identified gaps in 
environmental regulations on its reserve. While federal laws will 
continue to apply on reserves under the FNLMA, the First Nation 
must develop environmental protection standards and punishments 
that have at least the same effect as the laws in the province in which 
it is located. This could mean that the First Nation must develop and 
implement regulations where the federal government has yet to do so. 


6.73 ‘To develop such regulations, INAC, in conjunction with the 
First Nation and Environment Canada, must first develop and sign an 
environmental management agreement. Although some work has 
begun, no such agreements had been completed by the end of our 
audit; therefore, none of the 22 First Nations with land codes had 
developed any environmental regulations. Although First Nations with 
land codes assume increased responsibilities for environmental 
management on reserves, without an environmental management 
agreement they may not have the means and capacity to meet these 
additional responsibilities. 


6.74 The federal government has the authority to regulate 
environmental threats on reserves. Environment Canada has 
conducted 50 inspections on reserves in the last three years related to 
regulations under the Canadian Environmental Protection Act, 1999, and 
the Fisheries Act. INAC informed us that it was not systematically 
conducting inspections related to the Indian Reserve Waste Disposal 
Regulations under the Indian Act. Both departments have also left some 
other environmental threats unregulated. Passing these responsibilities 
to First Nations under the FNLMA may not address the risks. This 
leaves many communities on reserves exposed to conditions that other 
communities are protected from by regulation and enforcement. 


6.75 Recommendation. Indian and Northern Affairs Canada and 
Environment Canada should work together, in partnership with First 
Nations, to develop and implement a strategy to identify and address 
residual environmental regulatory gaps on reserves, as required. 


The departments’ response. Agreed. Indian and Northern Affairs 
Canada and Environment Canada will work together, and in 
partnership with First Nations, to develop a strategy to identify and 
close residual regulatory environmental gaps on reserves. However, 
some of these gaps will require legislative changes. 
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Non-regulatory management measures have not worked as planned 


6.76 Aside from regulations, INAC and Environment Canada have 
other means of managing environmental threats. We assessed two 
means that INAC uses to address environmental threats on reserves. 


6.77 Environmental provisions in leases to address regulatory gaps. 
Over half of First Nations with reserves remain subject to all the land 
management provisions of the Indian Act. For these First Nations, 
INAC retains full control and responsibility for fulfilling the 
requirements of the Indian Act, such as leasing land for commercial 
developments like hotels, gas stations, and stores. For new leases, the 
Department’s land management manual commits the Department to 
ensuring that the condition of the land is evaluated and, if required by 
law, that an environmental assessment is completed. 


6.78 ‘To determine whether INAC had met these requirements, we 
looked at a representative sample of 51 of the 936 leases on reserve 
lands in British Columbia, Saskatchewan, and Ontario that were 
processed under the Indian Act over the past three years. 


6.79 We found that in 60 percent of our sample, the Department did 
not obtain the required environmental information prior to signing or 
renewing leases. It therefore might have signed leases with parties 
without having completed the proper environmental reviews. 


6.80 A standard provision we found in leases states that lessees must 
respect all applicable environmental laws, including those of provincial 
governments. Since provincial land management laws do not apply to 
reserves and there are few federal regulations governing environmental 
protection on reserves, it is questionable whether a lease clause 
requiring compliance with applicable environmental regulations 
provides any increased protection for the environment or human 


health. 
6.81 Gsniweaused site liabilities. As of April 2008, INAC had 


identified 1,610 contaminated sites on reserves, mostly related to 
abandoned dumps and fuel spills. Of these, 557 had been classified as 
high-risk or medium-risk sites and therefore priority sites for cleanup or 
risk management. Moreover, the number of identified contaminated 
sites requiring investigation and possible cleanup has been increasing, 
as INAC continues to add suspected sites to its list. Between 

April 2008 and April 2009, the Department identified over 270 more 
suspected contaminated sites on reserves. Its financial liability for 
contaminated sites on reserves was projected at about $143 million as 


of April 2009, up from $98 million in April 2008. 
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6.82 The federal government has a $3.5 billion cost-shared program 
to help federal departments address contaminated sites under their 
responsibility if the contamination occurred before April 1998. The 
government’s objectives are to reduce environmental and human 
health risks and to eliminate liabilities from these sites by 2020. In the 
2008-09 fiscal year, the Department used $10.8 million from this fund 
to clean up or manage risks of 58 on-reserve sites. The Department has 
been identifying additional contaminated sites on reserves more 
quickly than it has been addressing them. 


6.83 A 2008 evaluation by INAC of its contaminated sites program 
concluded that the Department will not meet the federal government’s 
2020 target to eliminate known liability related to sites that became 
contaminated prior to 1998. According to the evaluation, none of the 
confirmed high-risk or medium-risk sites on reserves had been fully 
remediated and no actions had been taken for two thirds of these sites 
to mitigate the potential risks to health and the environment. The 
INAC evaluation also raised concerns about the lack of regulatory and 
enforcement tools to stop the creation of additional contaminated sites 
on reserves. 


6.84. Recommendation. Indian and Northern Affairs Canada should 
develop and implement a plan to remediate, by 2020, high-risk and 
medium-risk sites on reserves that became contaminated prior to 1998, 
as the federal government has committed to doing. 


The Department’s response. Agreed. The Indian and Inuit Affairs 
Contaminated Sites Management Program of Indian and Northern 
Affairs Canada will continue to work with First Nation communities to 
assess suspected and known contaminated sites, with the goal of 
reducing risks to human health and the environment and reducing 
departmental liabilities. 


The Contaminated Sites Management Program is currently on target 
to meet the 2020 deadline by maintaining an annual 15 percent 
reduction in the departmental liability related to known contaminated 
sites. As the program does not have dedicated program funds and has 
in the past been supported in part through internal reallocation, 
success in meeting 2020 targets will depend primarily on future levels 
of funding for the Federal Contaminated Sites Action Plan. 
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Funding constraints 


Insufficient funding has been committed to meet program requirements 


6.85 Throughout our audit, we found several instances where the 
federal government had not met its commitments and obligations. 
The reasons cited by department officials often related to the lack of 
an identified source of funding to meet commitments. 


6.86 Reserve Land and Environment Management Program 
(RLEMP). We found that this initiative was not available to the 
majority of First Nations and that training was limited to one 
individual for each First Nation accepted into the program. 
Department officials explained that the RLEMP is still a pilot program 
and is not more widely available to First Nations due to a lack of 
funding. Department officials also noted that the pilot prograrn is due 
to expire on 31 March 2010, At the end of our audit, Indian and 
Northern Affairs Canada was seeking new funding to make the 
program more widely available. 


6.87 First Nations Land Management Act (FNLMA). The FNLMA 
was proposed with a funding envelope of more than $100 million over 
its first five years. Department officials explained that they had been 
reallocating about $16 million per year from other approved programs 
but this had been insufficient to address all of the Department’s 
responsibilities under this regime. Department officials told us that 
until dedicated funds are made available, the Department is no longer 
accepting additional First Nations into the FNLMA regime. 


6.88 Environmental management on reserves. We found that gaps 
remain both in the regulations that apply on reserves compared with 
those off reserves and in the implementation of environmental 
management actions. INAC officials cited the lack of capacity and 
resources as among the main inhibitors to enforcing the Indian Reserve 
Waste Disposal Regulations and implementing the Department’s 
Environmental Stewardship Strategy. 


6.89 ‘To work with First Nations under the FNLMA, Environment 
Canada and INAC signed a five-year memorandum of understanding 
(MOU) that identified roles and responsibilities for each department 
to develop environmental management agreements with First Nations 
that have a land code. Environment Canada has completed some 
activities under the MOU but withdrew from further participation 
after less than two years, evoking a provision under the MOU that 
allows them to withdraw if the required resources have not been 
secured. 
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6.90 Recommendation. Indian and Northern Affairs Canada and 
Environment Canada should each assess the funding requirements 
necessary to fulfill their land management responsibilities on reserves 
and meet their commitments. Each department should also identify 
how it will finance these responsibilities. 


Indian and Northern Affairs Canada’s response. Agreed. 

The Department is regularly assessing its funding requirements to 
fulfill land management responsibilities on reserves. To fully implement 
the First Nations Land Management Act regime for current participants 
and to expand it to a significant number of new participants, the 
Department would require additional funding in future years. 


Environment Canada’s response. Agreed. The Department is 
assessing funding requirements for negotiating environmental 
management agreements under the First Nations Land Management 
Act, and options to address these requirements will be explored. 
Funding for other land management responsibilities will be assessed as 
part of the normal annual planning and priority-setting conducted by 
the Department. 


Conclusion 


6.91 Indian and Northern Affairs Canada (INAC) has developed 
legislative and program measures to support key elements of land 
management on reserves. It has provided greater control to First 
Nations to enable them to address economic development challenges 
on reserve lands. 


6.92 The First Nations Land Management Act (FNLMA) is intended to 
enable First Nations to make timely business and administrative 
decisions and to accelerate their progress in economic development, 
resources management, and land use planning. However, less than a 
quarter of the First Nations who were expected to take advantage of 
FNLMA programming have been able to do so. INAC has been unable 


to fund the full implementation of this program. 


6.93 INAC and Environment Canada have not addressed significant 
gaps in the regulatory framework that protects reserve lands from 
environmental threats. Provincial and municipal environmental 
regulations and zoning laws that protect communities off reserves do 
not apply to reserve lands. 


6.94 INAC has not met key responsibilities for implementing existing 
regulations to provide essential environmental protection on First 
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Nations reserves. While regulations exist under the Indian Act to cover 
solid waste management, most landfill sites on reserves operate 
without permits, monitoring, or enforcement by INAC. Similarly, while 
these same regulations apply to liquid waste, INAC does not issue 
permits or monitor and enforce compliance with the regulations for 
sewage treatment and disposal. 


6.95 INAC and Environment Canada have not established a 
regulatory regime that protects the environment of First Nations 
reserves and that is comparable to the regime provided by provincial 
laws. They have not filled the regulatory void that occurs when 
provincial laws governing land management do not apply on reserves; 
nor have they adequately managed the environmental threats that this 
void creates. 
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About the Audit 


All of the audit work in this chapter was conducted in accordance with the standards for assurance 
engagements set by The Canadian Institute of Chartered Accountants. While the Office adopts these 
standards as the minimum requirement for our audits, we also draw upon the standards and practices 
of other disciplines. 


Objectives 


The objectives of the audit were to determine whether Indian and Northern Affairs Canada (INAC) 
and Environment Canada 


- developed legislative options, programs, and policies in support of First Nations land management 
and environmental protection; and 


- met key responsibilities for implementing federal acts, regulations, programs, and/or policies in support 
of First Nations land management and provided essential environmental protection of First Nations 
reserves. 


Scope and approach 


The entities included in this audit are INAC and Environment Canada—primarily, the Lands and 
Economic Development Sector of INAC and the Environmental Stewardship Branch of Environment 
Canada. We obtained and assessed related documentation and interviewed department officials and land 
managers in the regional offices. 


While First Nations were not subject to audit, we visited selected First Nations and First Nations 
organizations to obtain their perspectives on land management. First Nations with self-governance 
agreements and comprehensive land claim agreements were not included in this audit. 


We reviewed the three main land management regimes accessible to First Nations: The Indian Act, 
the Reserve Land and Environment Management Program, and the First Nations Land Management Act. 


We selected three regions in which to conduct our regional audit work: British Columbia, Saskatchewan, 
and Ontario. These regions had the greatest number of registered land transactions over the past 
three years. 


Environmental management regulatory gaps on reserves cover a broad spectrum. The audit focused on 
areas identified as essential for First Nations in the Framework Agreement on First Nation Land 
Management: solid waste management, fuel storage tank management, sewage treatment and disposal, 
and environmental emergencies. It also focused on the legislation of the federal and Ontario governments 


most closely related to regulating environmental threats. 


For Exhibit 6.2, we examined Ontario regulations related to the province’s Environmental Protection Act, 
the Ontario Water Resources Act, the Technical Standards and Safety Act, the Building Code Act, and the 
Dangerous Goods Transportation Act. We compared the regulatory coverage these provided with regulations 
applicable to reserves from the federal Indian Act, Canadian Environmental Protection Act, 1999, the 
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Fisheries Act, the Transportation of Dangerous Goods Act, and the Indian Oil and Gas Act. Regulations from 
most of these federal acts are equally applicable on and off reserves. 


Criteria 


Listed below are the criteria that were used to conduct this audit and their sources. 


Criteria 


We expected that Indian and Northern Affairs Canada (INAC) 
would have developed legislative options, programs, and policies 
to support the management of First Nations lands. 


~ Sources 
Constitution Act, 1982, section 91(24) 


Department of Indian Affairs and Northern Development Act, 
section 4(a) 


Indian and Northern Affairs Canada’s mandate (INAC website) 


2009-2010 Indian and Northern Affairs Canada, Program 
Activity Architecture: Strategic Outcomes—The Land 


We expected that INAC would have developed legislative options, 
programs, and policies to transfer control over land management 
to First Nations who request it. 


Constitution Act, 1982, section 91(24) 


Department of Indian Affairs and Northern Development Act, 
section 4 


Indian Act, section 53(1) 
INAC Land Management Manual, Chapter 11 
INAC Report on Plans and Priorities 2006-07 


We expected that INAC and Environment Canada would have 
developed legislative options, programs, and policies related to 
solid waste management, fuel storage tank management, sewage 
treatment and disposal, and environmental emergencies 
comparable with those off reserves. 


Canadian Environmental Assessment Act, section 4(1)(c) 
Canadian Environmental Protection Act, 1999, Part 9 
INAC’s Sustainable Development Strategy 2004-2006 
INAC’s Environmental Stewardship Strategy 

INAC’s Program Activity Architecture 


We expected that INAC would have met its key responsibilities in 
processing land transactions under the /ndian Act. 


Indian and Northern Affairs Canada, Land Management 
Manual, chapters 7, 8, 11, and 12 


INAC’s mandate (website) 


“We expected that INAC would have transferred control of land 
management to First Nations who requested it. 


Indian Act, sections 53(1) and 60(1) 
First Nations Land Management Act, section 6(1) 


Reserve Land and Environment.Management Program, 
Backgrounder (INAC website) 


Indian and Northern Affairs Canada, Land Management 
Manual, Chapter 11 


INAC Land Management Program, Goals (website) 


INAC Report on Plans and Priorities 2006-07 
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ron : Sat Sources 
ladian Act, section 60(1) 
Indian and Northern Affairs Canada, Land Management 
Manual, Chapter 11 

INAC Land Management,. Description of Operations (website) 
INAC Report on Plans and Priorities, 2006-07 and 2008-09 
INAC’s Program Activity Architecture 


Framework Agreement on First Nation Land Management, 
paras. 21 and 39.1(e) 


We expected that INAC and Environment Canada would have 
met their key responsibilities in implementing their legislative 
options, programs, and policies related to solid waste 
management, fuel storage tank management, sewage treatment 
and disposal, and environmental emergencies in place on 
reserves. 


Department of Indian Affairs and Northern Development Act, 
section 4 


Canadian Environmental Protection Act, 1999, Part 1, 
section 9; Part 3, sections 44.1(a) and 217; and Part 9, 
sections 208 and 209 


Indian Reserve Waste Disposal Regulations 


Environmental Stewardship Strategy 


Management reviewed and accepted the suitability of the criteria used in the audit. 


Period covered by the audit 


We examined legislative options, institutions, and programs developed between March 2005 and 
May 2009. We examined actions by Indian and Northern Affairs Canada and Environment Canada to 


transfer control over land management to First Nations and address environmental threats on reserves 


over the past three years. 


Audit work for this chapter was substantially completed on 29 May 2009. 


Audit team 


Assistant Auditor General:-Ronnie Campbell 


Principal: Frank Barrett 
Lead Auditor: Nadine Cormier 
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Alexandre Boucher 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 6. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


Transfer of control of land management 


6.34 Indian and Northern Affairs: The Department’s response. Agreed. The Reserve Land and 
Canada should ensure that First Environment Management Program (RLEMP) began as a pilot 
Nations who are ready and who want program in 2005 and has been funded internally since its 
greater autonomy over their land inception. Existing authorities were used to allow for a pilot of 
management can access either the the RLEMP limited to First Nations participating in existing 
Reserve Land and Environment programs. 7a. 


Management Program or the First 
Nations Land Management Act regime. 
(6.14-6.33) 


Since 1996, a number of First Nations have sought access to the 
First Nations Land Management Act (FNLMA) regime. In 2002, 
an amendment to the Framework Agreement on First Nation 
Land Management was made to allow up to 30 First Nations in 
the developmental phase of the FNLMA regime at any given 
time. . 


In March 2008, INAC had to close the FNLMA regime to new 
entrants due to a lack of funding. The Department is currently 
seeking additional resources to effectively implement the 
FNLMA regime for participants and expand it to new First 
Nations. - 


In summer 2009, a sustainable funding source for the RLEMP 
was secured through the Aboriginal Economic Development 
Action Plan. With dedicated funding over four years, INAC will 
now be able to fully implement this program and allow up to 

20 new First Nations per year to enter the RLEMP 
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6.47 Indian and Northern Affairs 
Canada, in collaboration with First 
Nations, should seek to ensure that an 
appropriate training program is 
developed for First Nations in the First 
Nations Land Management Act regime. 
The Department should also ensure 
that appropriate training is made 
available to First Nations in either the 
Reserve Land and Environment 
Management Program or the FNLMA 
regime. This training should address the 
requirements under each of the 
respective regimes. (6.35-6.46) 
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Agreed. The Reserve Land and Environment Management 
Program (RLEMP) has provided the only professional land 
management training available to First Nations for the past four 
years. Program evaluations and reviews of the RLEMP training 
program have concluded that the design and delivery of the 
training program are sound, credible, relevant and responsive to 
the needs of First Nations land managers and their communities. 


From 1999 to 2005, Indian and Northern Affairs Canada 
(INAC) provided funding to the First Nations Land 
Management Resource Centre to develop a training strategy and 
curricula that would address the specific capacity requirements 
of First Nations Land Management First Nations. In 2005, 
INAC discontinued this funding due to a lack of progress. While 


‘the RLEMP training was designed for First Nations managing 


lands under the Indian Act, the Department decided to offer that 
training on a priority basis to First Nations Land Management 
First Nations as a temporary measure. 


A renewed effort by the Resource Centre to develop training 
and curricula specific to First Nations Land Management is 
currently being undertaken and INAC anticipates receiving a 
proposal to that effect in 2009-2010. 


The RLEMP pilot project has been funded internally since its 
inception in 2005. The necessary sources of funds have not 
always been identified or available at the beginning of each fiscal 
year. As a result, there has been a delay in confirming whether 
the RLEMP training program could be offered in a given year 
and in notifying students of their acceptance into the training 
program. 


As mentioned previously, a sustainable funding source for the 
RLEMP was secured in spring 2009 through the Aboriginal 
Economic Development Action Plan. With dedicated funding 
for this program over four years, INAC will now be able to 
engage in the appropriate planning and forecasting required to 
notify new students slated to participate in the RLEMP training 
program much sooner than before. 
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Environmental protection 


6.75 Indian and Northern Affairs 
Canada and Environment Canada 
should work together, in partnership 
with First Nations, to develop and 
implement a strategy to identify and 
address residual environmental 


regulatory gaps on reserves, as required. 


(6.48-6.74) 


6.84 Indian and Northern Affairs 


Canada should develop and implement 


a plan to remediate, by 2020, high-risk 
and medium-risk sites on reserves that 
became contaminated prior to 1998, as 
the federal government has committed 


to doing. (6.76—6.83) 


Funding constraints 


6.90 Indian and Northern Affairs 
Canada and Environment Canada 
should each assess the funding _ 
requirements necessary to fulfill their 
land management responsibilities on 
reserves and meet their commitments. 
Each department should also identify 
how it will finance these 


responsibilities. (6.85-6.89) 


Agreed. Indian and Northern Affairs Canada and Environment 
Canada will work together, and in partnership with First 
Nations, to develop a strategy to identify and close residual 
regulatory environmental gaps on reserves. However, some of 
these gaps will require legislative changes. 


Agreed. The Indian and Inuit Affairs Contaminated Sites 
Management Program of Indian and Northern Affairs Canada 
will continue to work with First Nation communities to assess 
suspected and known contaminated sites with the goal of 
reducing risks to human health and the environment and 
reducing departmental liabilities. 


The Contaminated Sites Management Program is currently on 
target to meet the 2020 deadline by maintaining an annual 15 
percent reduction in the departmental liability related to known 
contaminated sites. As the program does not have dedicated 
program funds and has in the past been supported in part 
through internal reallocation, success in meeting 2020 targets 
will depend primarily on future levels of funding for the Federal 
Contaminated Sites Action Plan. . 


Indian and Northern Affairs Canada’s response. Agreed. 
The Department is regularly assessing its funding requirements 
to fulfill land management responsibilities on reserves. To fully 
implement the First Nations Land Management Act regime for 
current participants and to expand it to a significant number of 
new participants, the Department would require additional 
funding in future years. 


Environment Canada’s response. Agreed. The Department is 
assessing funding requirements for negotiating environmental 
management agreements under the First Nations Land 
Management Act, and options to address these requirements will 
be explored. Funding for other land management responsibilities 
will be assessed as part of the normal annual planning and 
priority-setting conducted by the Department. 
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Cyber security has recently received more attention, but significant challenges remain 


Conclusion 
About the Audit 


Appendix 


List of recommendations 


26 


29 
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What we examined 


Emergency Management— 


Public Safety Canada 


Main Points 


Emergency management refers to a wide range of measures to protect 
communities and the environment from risks and to recover from 
emergency events stemming from either natural or human-induced 
causes. While some emergencies in Canada can:be handled locally by 
municipalities or provinces, the federal government will assist when 
requested, when the emergency transcends jurisdictional boundaries, 
or when its assistance is in the national interest. As emergency events 
today can escalate quickly, this federal capability has become 
increasingly necessary. 


Through legislation and government policy, Public Safety Canada, 
which was created in December 2003, is responsible for leading by 
coordinating the management of emergencies among federal 
departments and agencies. This includes establishing policies and 
programs for the preparation, testing and exercising, and implementing 
emergency management plans; it also includes monitoring and 
coordinating a common federal approach to emergency response along 
with the provinces—an “all-hazards” approach incorporating 
prevention and mitigation, preparedness, response, and recovery. The 
Department’s responsibility for emergency management includes 
coordinating the protection of critical infrastructure—from planning 
for emergencies to recovering from them. Critical infrastructure 
includes physical and information technology facilities, networks, 
services, and assets essential to the health and safety or economic 
well-being of Canadians. 


We examined how Public Safety Canada carries out these 
responsibilities. In addition, we looked at its efforts to enhance 
emergency response and recovery in coordination with six other 
departments that have specific roles in emergency management. Our 
audit included assessing the government’s progress on some of the 
commitments it made to Parliament. Our audit covers performance of 
federal departments and agencies and events taking place since our 


last audit, reported in April 2005, and 15 June 2009. 
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2 Chapter 7 


What we found 


We did not examine the performance of emergency management 
efforts by provinces, territories, or local communities. 


The HINI1 pandemic, the 2003 eastern seaboard power blackout, 
Severe Acute Respiratory Syndrome (SARS), massive flooding, and 
terrorist conspiracies and attacks have demonstrated that global trade, 
international travel, and cyberspace have increased the speed at which 
emergencies escalate in scope and severity. Today, many emergencies 
can be difficult to contain by a single government department or 
jurisdiction. A federal response is needed for emergencies that are 
beyond the capacities of other players—emergencies that may have a 
low probability of occurrence but a high potential impact. 


Public Safety Canada is faced with the challenging task of providing 
the coordination necessary for an overall federal approach to 
emergency management, in an environment where departments have 
operated as needed and through their ministers to provide federal 
assistance on a case-by-case basis. 


¢ Public Safety Canada has not exercised the leadership necessary to 
coordinate emergency management activities, including critical 
infrastructure protection in Canada. For example, it has yet to 
develop the policies and programs that would help clarify its 
leadership and coordination role for an “all-hazards” approach to the 
emergency management activities of departments. Public Safety 
Canada has taken the first step by developing the interim Federal 
Emergency Response Plan, which it considers to be final although it 
has not been formally approved by the government. Nor does the 
Plan include updated or completed definitions of the roles, 
responsibilities, and capabilities needed for an integrated, 
coordinated approach to emergency response. 


¢ Public Safety Canada has made considerable progress in improving 
federal emergency coordination through its Government Operations 
Centre. It keeps other departments informed of the status of events 
on a real-time basis and also produces regular situation awareness 
reports for such issues as the HIN1 virus, which allows decisions to 
be based on a common set of facts. 


- Public Safety Canada has developed a strategy to protect Canada’s 
critical infrastructure, but this strategy is still in draft form. At the 
time of our audit, the critical infrastructure that needs to be 
protected had not yet been determined. Public Safety has moved 
forward in promoting a consistent approach to protection efforts 
across government. For example, it has categorized critical 
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infrastructure into 10 sectors, each headed by a federal department. 
However, it has not provided those departments with guidance for 
determining what assets or facilities are critical and require 
protection. 


¢ Progress has been slow until 2009 on Public Safety Canada’s 2004 
commitment to develop a cyber security strategy, although threats to 
computer-based critical infrastructure, including federal information 
systems, have been growing and evolving. To date, it has identified 
the key elements of a cyber strategy and initiated action on a list of 
current cyber security initiatives along with other federal 
government departments. However, at the time of our audit, no 
date was planned for obtaining formal approval of the strategy. 


« Although the 2004 National Security Policy called for first 
responders’ equipment and communications to be interoperable, 
key gaps remain for voice communications. This limits the ability of 
fire, police, and ambulance services to work together and with other 
jurisdictions in an emergency. The Department has directed little or 
no funding toward standardizing equipment. 


The Department and the Privy Council Office have responded. The 
Department and the Privy Council Office agree with all of the 
recommendations that are addressed to them. Their detailed responses 
follow the recommendations throughout the chapter. 
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EMERGENCY MANAGEMENT—PUBLIC SAFETY CANADA 


Introduction 


7.1 Emergencies today can have a broader impact than those of the 
past. Examples of emergencies that have recently affected Canadians 
include the outbreaks of HIN1 and avian influenza, severe acute 
respiratory syndrome (SARS), listeriosis, and mad cow disease; 

the 1998 ice storm in Eastern Canada; and the 2003 power blackout 
across the eastern seaboard. Urban density, international travel, and 
global trade have increased the speed at which emergencies can 
escalate and spread. Today, many emergencies can be difficult to 
contain, and the impact is likely to be greater. A federal response is 
needed for those emergencies that are beyond the capacity of 
municipalities or individual provinces or territories—emergencies that 
may have a low probability of occurrence but can have a high potential 
impact. To be able to respond effectively to large-scale emergencies 
and reduce the potential loss of life and property damage, there needs 
to be extensive planning and coordination. 


7.2. Under Canada’s Constitution Act, 1867, provinces and territories 
have primary responsibility for emergency management within their 
boundaries. Emergencies such as fires and floods may remain local in 
nature and, if so, may be effectively managed within the local resources 
of the municipality and province or territory. If an incident escalates, 
so do the response activities of various levels of government. At the 
request of a province or territory or where the type of emergency falls 
within federal jurisdiction or occurs on federal lands, the federal 
government provides help to manage and coordinate the response to 
an emergency. The Emergency Management Act (2007) established that 
the Department of Public Safety and Emergency Preparedness (Public 
Safety Canada) is responsible for responding to requests for assistance 
made by provinces and territories and for coordinating the assistance 
provided by other federal departments and agencies to the provinces 
and territories. 


73 Following the events of September 11, 2001, the Canadian 
government changed its approach to emergency preparedness and 
response. At that time, there was a highly decentralized division of 
responsibilities among federal departments, provinces, and territories. 
In December 2003, the government created the Department of Public 
Safety and Emergency Preparedness, bringing together emergency 
preparedness, national security, and policing responsibilities within one 
federal department. This restructuring was intended to better integrate 
public safety efforts and link various federal programs more closely. 
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7.4 Building the capability to manage a coordinated federal response 
to an emergency of national significance is a huge undertaking and 
cannot be achieved overnight. In the past, federal departments had 
organized their emergency response actions as situations arose. 
However, given the changing nature of national emergencies, this is no 
longer sufficient. Recognizing this, the federal government issued the 
National Security Policy in April 2004, which called for the federal 
government to be prepared to play an enhanced role in modern 
emergency management and to improve collaboration among 
governments and other entities. The 2004 policy outlined a number of 
initiatives to enhance the safety and security of Canadians. It 
identified the need for an “all-hazards” approach, meaning that 
whether or not the cause of an emergency is malicious, accidental, or 
natural, the federal government would be prepared to respond. To. 
facilitate this, the policy called for an updated emergency response 
system in which federal entities would work together in a coordinated 
manner. As well, it identified the need for federal departments and 
agencies to be more strongly linked with emergency operations at the 
provincial, territorial, and local levels. 


7.5 The 2004 National Security Policy, our 2005 audit of national 
security and emergency preparedness, the House of Commons 
Standing Committee on Public Accounts, and the Senate Standing 
Committee on National Security and Defence all called for updated 
federal legislation to clearly define and ensure adequate emergency 
management powers and responsibilities for the Minister of Public 


Safety. 


7.6 In 2005, the Department of Public Safety and Emergency 
Preparedness Act was passed. It stipulates that the Minister of Public 
Safety is to exercise “leadership at the national level relating to public 
safety and emergency preparedness.” When she appeared before the 
Standing Committee on Justice, Human Rights, Public Safety and 
Emergency Preparedness, the Minister explained that she would be 
responsible for coordinating the federal response to emergencies, while 
respecting the Prime Minister’s prerogative in matters relating to 
national security and to the statutory authorities of other ministers. 


7.7 In January 2007, federal, provincial, and territorial ministers 
agreed that emergency management would adopt a comprehensive 
all-hazards approach. This approach would incorporate the 

four functions of emergency management: prevention and mitigation, 
preparedness, response, and recovery. 
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78 In August 2007, the Emergency Management Act came into force. 
It assigns to the Minister-of Public Safety the responsibility to “exercise 
leadership relating to emergency management by coordinating federal 
emergency management activities” (Exhibit 7.1). 


Exhibit 7.1 The Emergency Management Act dictates specific responsibilities for ministers 


The Emergency Management Act requires the Minister of Public Safety to exercise 
leadership for emergency management by coordinating emergency management 
activities among federal departments and agencies, and in cooperation with the 
provinces and territories. 


The Minister’s responsibilities include 


¢ establishing policies and programs, and providing advice to other departments 
for the preparation of their emergency management plans; 


* analyzing and evaluating emergency management plans prepared by federal 
entities; 

* monitoring potential and actual emergencies and coordinating the federal 
response to an emergency; 


* coordinating federal emergency management activities with those of the 
provinces, and through the provinces, those of local authorities; 


* coordinating the provision of assistance to a province; 


* promoting a common approach to emergency management, including 
the adoption of standards and best practices; and 


e conducting exercises and providing emergency management education 
and training. 


As well, other federal ministers are to identify the risks that are within their area of 
responsibility, including those related to critical infrastructure, and to prepare, 
maintain, test, implement, and exercise emergency management plans in respect of 
those risks in compliance with the policies, programs, and other measures established 
by the Minister of Public Safety. 


Source: Adapted from the Emergency Management Act 


7.9 Public Safety Canada is the coordinating agency for federal 
departments, which have various roles to play in an emergency. Public 
Safety Canada is to ensure that the federal government is ready to 
respond to any future emergencies through the development of 
policies, standards, and plans that define roles and responsibilities. 
The aim is to eliminate the potential for confusion when responding 
in a crisis and provide a federal point for coordination. 


7.10 Ifa department or agency has a clear mandate to respond to an 
emergency and is responsible to act, it is the subject matter expert. 
However, if emergencies escalate and spread, other federal 
departments may be required to play a role to manage the impact 
within their area of expertise. For example, for an incident involving a 
terrorist or criminal act, the Royal Canadian Mounted Police (RCMP) 
would be the primary federal response agency in its law enforcement 
role. For a natural disaster involving an earthquake or a power outage, 
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Natural Resources Canada would be the primary subject matter expert. 
The Public Health Agency of Canada would be the subject matter 
expert for public health, including infectious diseases, in concert with 
Health Canada. Other departments or agencies, such as the Canada 
Border Services Agency, would play a supporting role. In each of these 
examples, Public Safety Canada plays a coordinating role in helping to 
receive information and communicate the current situation to other 
departments and agencies, and to senior officials in the federal 
government and other jurisdictions. 


7.11 For emergency management, Public Safety Canada had a budget 
of $58.5 million and 400 employees for the 2008—09 fiscal year. Many 
of Public Safety Canada’s emergency management programs are 
delivered through 11 regional offices. It also manages the federal 
Government Operations Centre that monitors emerging threats and 
provides round-the-clock coordination and support to government 
entities in the event of a national emergency. As well, it oversees the 
conduct of exercises on emergency management at the national level 
and an inter-jurisdictional training program for local frontline 
emergency workers at its Canadian Emergency Management College. 


7.12 Public Safety Canada’s role as the lead department for 
coordinating federal emergency management includes critical 
infrastructure protection. Critical infrastructure consists of physical and 
information technology facilities, networks, services, and assets essential 
to the health and safety or economic well-being of Canadians, and the 
effective functioning of government. Examples of critical infrastructure 
include food, water, and energy supplies; health services; financial 
systems; and communication networks, including the Internet. Events 
such as the 1998 ice storm in Eastern Canada and the 2003 power 
blackout across the eastern seaboard highlight the impact of the failure 
of the electrical grid. The vast majority of Canada’s critical infrastructure 
is owned by the private sector or managed through another level of 
government. This creates a challenge for the federal government to 
establish its role with owners and operators and thereby ensure the 
protection and resiliency of the nation’s critical infrastructure. 


Focus of the audit 


7.13 In this audit, we focused on four main responsibilities of Public 
Safety Canada: 


¢ To establish policies and programs for emergency management 
plans and operations, provide advice to departments, and evaluate 
their plans. 
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¢ To coordinate the emergency management activities among 
federal government institutions along with those of the provinces 
and territories. 


¢ ‘Io promote a common approach to emergency management, 
including the adoption of standards and best practices. 


¢ To coordinate the protection of Canada’s critical infrastructure. 


7.14 Specifically, we examined Public Safety Canada’s responsibility to 
lead by coordinating the efforts of other federal entities and by 
coordinating federal efforts with those of the provinces and territories. 
We focused mainly on the Department’s preparedness efforts, 
including its coordination of the provision of critical infrastructure 
protection. As well, we examined progress by Public Safety Canada in 
enhancing emergency response and recovery in coordination with 
government departments and agencies. 


7.15 We did not examine the performance of provinces, territories, or 
local communities in their delivery of emergency management services 
or activities, nor did we examine provincial and territorial or private 
sector critical infrastructure protection efforts. We also did not 
examine the security activities carried out in preparation for the 2010 
Olympic and Paralympic Games, as responsibility for these activities 
was assigned to the Office of the Coordinator for 2010 Olympics and 
G8 Security, which reports to the National Security Advisor. 


7.16 More details on the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


Establishing and exercising federal leadership has been a challenge 


7.17 Public Safety Canada is responsible under legislation to exercise 
leadership through planning, establishing policies and programs for 
emergency preparedness, cooperating with provinces and territories, 
and promoting a common approach to emergency management. It is 
responsible for coordinating the emergency management activities of 
various federal departments and agencies and fostering a cooperative 
approach to responding to emergencies. 


718 Because the subject matter expertise and experience for dealing 
with emergencies resides in several different departments, Public 
Safety Canada has an important role to ensure that all potential 
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hazards are addressed, that plans exist and have been shared and 
tested, and that, during a crisis, the kind of response needed is quickly 
established without confusion. However, Public Safety Canada does 
not assume control over other departments or tell them how to do 
their jobs. Each department remains responsible to its own minister 
and for acting as required under its own legislation. Public Safety 
Canada, under the Emergency Management Act, is responsible for 
establishing policies and programs that other ministers must follow in 
carrying out their emergency management responsibilities and 
determining how they will be coordinated. Given different mandates 
and accountabilities, it is important that Public Safety Canada know 
who it should communicate with and ensure that the various 
departments know how coordination will proceed and what the 
expected operating procedures will be. Nevertheless, each department 
determines whether it will assist during an emergency, what its role will 
be, and how it will operate with other federal, provincial, or territorial 
partners. 


7.19 We found that while Public Safety Canada played a coordination 
role in some emergencies, including participating in the development 
of response plans for avian and pandemic influenza, it has yet to 
establish the policies and programs that would help define its 
leadership and coordination role for emergency management in an 
all-hazards environment. Defining a leadership role when each 
department responds to its own ministerial direction, and coordinating 
that direction with other departments can be a challenge. 
Nevertheless, Public Safety Canada was established to address these 
concerns and determine how to coordinate and harmonize the 
activities of the different departments needed to deal with today’s 
complicated and broad-reaching situations. 


7.20 In order to move forward in its mandate to exercise leadership, 
Public Safety Canada needs to have experienced and knowledgeable 
staff in place. Another challenge we noted was that the Department 
has had difficulty attracting and retaining senior managers to provide 
the direction needed in its emergency management. This area of Public 
Safety Canada had an employee vacancy rate of 39 percent in 

the 2008-09 fiscal year and a vacancy rate of 50 percent the previous 
year. In April 2009, only 56 percent of senior managers had been in 
their jobs for more than 18 months. Turnover and change of staff has 
been particularly problematic, and in the 2008-09 fiscal year, the rate 
of employee movement (including appointments, promotions, 
deployments, acting assignments, and departures) was 71 percent in 
emergency management. 
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1.21 In 2006, Public Safety Canada was allocated approximately 
$115 million over five years to enhance its core capacity for emergency 
management; and, in 2008, it was allocated a further $28 million over 
five years. In the 2008-09 fiscal year, Public Safety Canada had an 
annual budget of $58.5 million for emergency management. However, 
it had not spent one third of its budget for emergency management in 
each of the past two years. In this context, it is evident that Public 
Safety Canada has been unable to develop its capacity for emergency 
management. 


7.22 In the face of these challenges, Public Safety Canada has taken 
the first steps toward establishing its leadership role by developing the 
interim Federal Emergency Response Plan, a framework for 
coordinating emergency response activities across government. Work 
has been under way on developing this plan, in various forms, 

since 2004. In June 2005, the House of Commons Standing 
Committee on Public Accounts recommended that Public Safety 
Canada obtain formal support for its plan from other departments. At 
the time of our audit, the Plan was still an outline of the requirements 
of an emergency response plan. The Plan has been presented to an 
interdepartmental committee of assistant deputy ministers. Although 
it has not been formally approved by Public Safety Canada or endorsed 
by other departments, officials told us that it is, nevertheless, being 
considered final. 


7.23. While the framework may be considered complete, the roles and 
responsibilities and the capabilities (contained in its annexes) needed 
for an integrated, coordinated approach to emergencies have not been 
updated or completed. Department officials told us that details on how 
the federal plan supports provincial and territorial plans and 
capabilities are being drafted. At the time of our audit, Public Safety 
Canada expected to share the draft document with provincial and 
territorial representatives in September 2009. While we recognize that 
the Federal Emergency Response Plan will always need to be updated 
to reflect changes in policies and practices, it is a significant policy 
document that, with formal government approval, would provide 
proper authority and clear support to Public Safety Canada. 


7.24. The Federal Emergency Response Plan outlines a 
decision-making process to help coordinate a federal response to 
emergencies. Since 2006, an interdepartmental assistant deputy 
ministers’ committee for emergency management (now, ADM-EMC) 
has met regularly to discuss emergency management priorities 

and to make decisions to guide federal government actions during 
emergencies. Depending on the severity of a situation, this committee 
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d 
may make decisions, or may refer the issue to the Federal Coordinating 


Officer (usually the Deputy Minister of Public Safety), who may refer 
the issue to a committee of deputy ministers. Similarly, the issue may 
be referred to Cabinet or, ultimately, to the Prime Minister. The federal 
emergency response structure is summarized in Exhibit 7.2. The 
ADM-EMC is co-chaired by Public Safety Canada and the Privy 
Council Office to facilitate the sharing of information should decisions 
need to be taken to a higher level. The ADM-EMC has served as the 
coordinating body for events such as the 2007 floods in British 
Columbia and the HIN1 virus pandemic in 2009. 


Exhibit 7.2 Information coordination and decision making for emergency response set out in the 
Federal Emergency Response Plan 


Cabinet Committee 


for Operations 


Deputy Ministers’ 
Committee 


_ Federal 
Coordinating Officer 


Decision making 


Assistant Deputy 
Ministers Emergency 
Management Committee 


Departmental 
national operations Government 
centres Operations Centre 


Provinces and 
territories 


Public Safety Canada and 
other government departments’ 


liaison/representatives International 


Information/Coordination 


Source: Adapted from the Federal Emergency Response Plan (April 2009) 


7.25 As part of our audit, we reviewed federal responses to 

six emergencies that occurred between August 2006 and May 2009, 
where multiple federal departments were involved and for which 
after-action reports were available. We tried to determine whether 
the Federal Emergency Response Plan was used as the framework 
for a coordinated response. In each of these cases, the Government 
Operations Centre was used to varying degrees to share information 
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and analysis among entities. However, the ADM-EMC, the body 
responsible for coordinating the federal response to an emergency, 

did not meet to discuss possible responses during three of these 

six emergencies. According to after-action reports prepared by 
participating departments for these emergencies, there were problems 
in coordinating the federal response among departments and agencies 
in all cases. Roles and responsibilities needed to achieve a coordinated 
approach were not well understood and some established practices 
were not followed. At the time of our audit, the ADM-EMC intended 


to clarify roles and responsibilities in its decision-making process. 


1.26 Recommendation. The Privy Council Office and Public Safety 
Canada should ensure that all components of the Federal Emergency 
Response Plan are completed and should obtain government approval 
for the plan. 


The Privy Council Office and the Department’s response. Agreed. 
The Privy Council Office and Public Safety Canada will seek approval 
for the completed Federal Emergency Response Plan (FERP) at the 
earliest possible date and the supporting Emergency Support Functions 
(ESFs) prior to the end of the 2009-10 fiscal year. Public Safety 
Canada will seek approval of the National Emergency Response 
System (NERS), an annex to the FERP which articulates how the 
FERP supports provincial and territorial emergency response plans, by 
the end of August 2010. Public Safety Canada will organize 
information sessions with departmental executive committees to brief 
departments on the FERP and their associated roles and 
responsibilities. The FERP and its components will be maintained as 
an evergreen document. ° 


A consistent risk management approach is lacking 


7.21 In order to be ready to respond, emergency management plans 
need to address the most important risks. In 2007, the Deputy 
Ministers’ Committee directed Public Safety Canada to assess the 
federal government's state of readiness for a national emergency. 
Through this review process, a number of capability gaps were 
identified; however, Public Safety Canada did not have a framework 
upon which to prioritize or rank the severity of the gaps and, as a 
result, has not moved forward with an action plan to address these 
gaps. As well, the review found that Public Safety Canada lacked an 
all-hazards risk assessment that identified potential hazards to public 
safety or security—whether malicious, natural, or accidental. It also 
lacked a framework to determine required capabilities to respond to 
these risks. 
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7.28 The 2004 National Security Policy and the 2007 Emergency 
Management Act recognized that the federal government needed to 
better understand Canada’s vulnerability to emerging risks and use this 
information to develop comprehensive emergency plans and programs. 
Under the Emergency Management Act, federal departments are to 
identify risks that are within their area of responsibility, and prepare 
emergency plans in respect of those risks according to the policies 
established by Public Safety Canada. Under its leadership role for 
emergency management activities, Public Safety Canada is to 
coordinate risk assessments in collaboration with other federal 
departments and to ensure that they have proper emergency 
management plans and preparedness measures in place. 


7.29 We found that Public Safety Canada has made limited progress 
in developing the guidance that departments need to achieve a 
consistent approach when identifying their risks and their emergency 
management plans and programs. A comprehensive risk and 
vulnerability assessment to guide the development of plans and 
response capabilities under an all-hazards approach has not been 
conducted in Canada. A Public Safety Canada study conducted 

in 2008 of 36 federal departments found wide variation in the risk 
assessment processes used by departments to guide the development of 
plans and capabilities. Some departments had no process in place. 

‘In the six federal departments we examined, we found that none had 
received any guidance from Public Safety Canada on conducting risk 
assessments for emergency planning, yet all of these departments were 
working to update their plans. Public Safety Canada initiated a project 
in April 2009 to streamline and validate these risk assessment 
processes for emergency planning and capabilities development. 

This project is in the preliminary planning stage. 


7.30 The Emergency Management Act stipulates that Public Safety 
Canada is responsible for reviewing departmental emergency 
management plans, which includes departmental business continuity 
plans. These plans are needed so that federal organizations can 
continue operating during an emergency. Under the Emergency 
Management Act, Public Safety Canada is responsible for ensuring that 
business continuity plans are complementary and meet the overall 
needs of the federal government. It had provided a self-assessment 
tool for departments to review their own business continuity plans. 
However, at the time of our audit, Public Safety Canada had not 
formally analyzed or evaluated departmental business continuity plans, 
nor did it have plans to do so. It had not determined whether there 
were gaps between departments. 
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1.31 Recommendation. As stipulated in the Emergency Management 
Act, Public Safety Canada should establish policies and programs and 
provide advice for departments to follow when identifying risks and 
developing their emergency management plans. 


The Department’s response. Agreed. In keeping with the all-hazards 
approach to emergency management, Public Safety Canada is leading 
the development of an Emergency Management Planning Framework 
that will provide departments and agencies with guidance, tools, and 
best practices for developing emergency management plans. It is also 
working with federal departments to develop an all-hazards risk 
assessment framework. Under the Emergency Management Act, it is 
the responsibility of each minister accountable to Parliament for a 
government institution to identify the risks that are within or related 
to his or her area of responsibility. 


7.32 Recommendation. As stipulated in the Emergency Management 
Act, Public Safety Canada should analyze and evaluate the emergency 
management plans prepared by departments to ensure that they are 
prepared according to the policies, programs, and advice provided, and 
it should identify potential gaps or risks to a coordinated emergency 
management response. 


The Department’s response. Agreed. Public Safety Canada is 
developing the Emergency Management Planning Framework, which 
will include performance measurements that will allow Public Safety 
Canada to analyze and evaluate emergency management plans 
produced by departments and agencies. The Framework will also 
include self-assessment tools for departments and agencies. Public 
Safety Canada is currently developing an approach to implement this 
initiative. 


There has been progress in developing a government operations centre 


7.33 In 2004, Public Safety Canada established the Government 
Operations Centre as the core of its federal coordination efforts for events 
of national significance. The role of the Government Operations Centre 
is not to act as a decision-making body in an emergency response, but 

to assemble and communicate information to decision makers. It is 
connected with the operations centres of 20 federal departments and 
agencies, as well as with those of the provinces and territories, and other 
countries, including the United States. . 


7.34. The Government Operations Centre has coordinated 
information and analysis among federal departments and provinces 
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for numerous events since its inception. The scope of the emergency 
determines the scale and extent of its functions. However, it has not 
clearly defined when or why its level of activation changes in response 
to the severity of events and what this means for participating 
departments. A government-wide exercise, conducted 

in February 2009 by Public Safety Canada, found that information 
analysis and sharing at the operations centre was poor. Furthermore, 
officials at Public Safety Canada told us that the Government 
Operations Centre did not have the physical facilities to support the 
number of staff needed to keep the operations centre fully functional 
for a major emergency lasting an extended period of time. Public Safety 
Canada was in the process of determining what corrective actions were 
needed as we completed our audit work. 


7.35 Public Safety Canada has made considerable progress in federal 
emergency coordination through its Government Operations Centre, 
as the centre operates on a continual basis and can track many 
potential or evolving events. It keeps other departments informed of 
the status of events on a real-time basis and alerts them if the events 
escalate into a more serious situation. The centre produced regular 
situation awareness reports for such issues as the HIN1 virus pandemic 
and Manitoba’s spring flooding in 2009, which allowed decisions to be 
based on a common set of facts. We noted that the Government 
Operations Centre reviewed how well it performed after events, but 
this was a verbal process. Results from these reviews are not normally 
tracked or monitored to ensure that corrective action is implemented. 


Lessons learned have not been used to improve emergency response 


7.36 In order for response plans to be reliable during an emergency, 
they must be regularly exercised, especially the plans for coordination 
between departments and agencies and between different levels of 
government. The National Security Policy and the Emergency 
Management Act call for regular exercises to assess the adequacy of 
emergency response plans in various scenarios. In 2004, the National 
Exercise Division was established within Public Safety Canada, with 
resources dedicated to staging regular national exercises at the federal, 
provincial, and municipal levels and consolidating lessons learned to 
improve future performance. 


7.37. Over the past three years, Public Safety Canada budgeted a total 
of $17.1 million to plan and conduct exercises related to emergency 
management across the federal government and with the provinces 
and municipalities, as well as to share lessons learned and best 
practices with exercise participants. However, over half of the budget 
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allocated to national exercises was not spent in each of the last 

three fiscal years. Public Safety Canada maintains a calendar that lists 
exercises planned among federal departments and has developed a 
framework for federal departments and agencies to coordinate their 
national exercise efforts. Since April 2005, Public Safety Canada has 
coordinated five federal exercises, shared in the coordination of 

eight multi-jurisdictional exercises, and participated in an additional 
two exercises. However, we found that exercises were designed to meet 
the training objectives of individual departments, rather than to test 
the government's overall coordination or readiness for a national 
emergency against identified risks. Public Safety Canada recognizes the 
need to increase the number of federal and multi-jurisdictional 
exercises. | 


7.38 In response to our April 2005 audit, Public Safety Canada 
committed to consolidating, on an ongoing basis, the results of lessons 
learned; however, at the time of our audit, it had not done so. The 
Department provided us with after-action reports for 14 of the 
exercises it coordinated or participated in since April 2005, but 
observations and recommendations from these reports were not 
systematically collected and used to improve emergency plans and 
operations. 


Coordination is unclear for responses to chemical, biological, radiological, nuclear, 
or explosives emergencies 


7.39 Following the events of September 11, 2001, Canada focused its 
attention on the significant threats posed by terrorist attacks and on 
the need to enhance readiness against emergencies caused by people, 
whether deliberate or accidental. In Budget 2001, the federal 
government allocated $513 million over six years to federal departments 
and agencies to improve their ability to respond to chemical, biological, 
radiological, or nuclear (CBRN) events, as these types of emergencies 
are beyond the response capacity of provinces, territories, and 
municipalities. Public Safety Canada receives $2.7 million annually for 
CBRN training. The initiative is currently being expanded to include 
the possibility of a threat due to explosives (CBRNE). 


7.40 We examined the status of efforts made to improve CBRNE 
response capability, where a coordinated and integrated approach 
among federal departments, as well as provincial and local 
jurisdictions, is essential to success. To enhance the capacity of local 
emergency workers to respond to a CBRNE event, Public Safety 
Canada leads a training program for first responders from municipal, 
provincial, and territorial governments, with a combined annual 
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federal budget of $12 million. From April 2003 to April 2009, it had 
trained 1,854 local first responders to assist during an event, and a 
further 10,400 had received awareness training. While Public Safety 
Canada.has administered participant questionnaires and consulted 
experts and other government departments, it has not conducted a 
formal needs analysis for its’ first responder training. | 


7.41 In 2002, a federal team was established to prepare for and 
respond to potential CBRNE events, combining the efforts of the 
RCMP and National Defence; at the time, Health Canada; and, 

since 2004, the Public Health Agency of Canada. We expected that 
Public Safety Canada would lead the efforts of these departments, and 
we looked for evidence of joint planning and execution to develop the 
capabilities needed for a coordinated response and recovery. 


7.42 Public Safety Canada is responsible for setting the overall federal 
policy on CBRNE issues. In 2005, it issued a federal strategy, 
identifying the roles and responsibilities of federal departments and 
agencies for an effective response to these types of emergencies. 
However, it did not address how federal departments and agencies 
would coordinate their resources with those of the provinces, 
territories, and municipalities to assist them in a national emergency, 
nor has it expanded the strategy to include explosives. At the time of 
our audit, Public Safety Canada was consulting with the provinces and 
territories to develop a national CBRNE strategy that included their 
responsibilities. 


7.43 While the current strategy states that the government is to take 
all possible measures to pre-empt, prevent, mitigate, and respond 
effectively to a potential CBRNE incident, it has not identified the 
desired capability, mandate, roles, or priorities for crisis or consequence 
management for the responsible federal organizations. The role of the 
federal CBRNE team is to manage the crisis phase of an emergency; 
however, the team does not have the resources to manage the after 
effects of a CBRNE incident, including assisting in mass casualty 
evacuation, medical aid, or decontamination. In August 2008, the 
three departments involved in the federal CBRNE response team 
informed Public Safety Canada of their concerns with the team’s 
mandate, capacity, training, and the compatibility of communications 
equipment. While the responsibilities of each team member were clear, 
there were no defined operational protocols or agreements on how the 
team would work together in a coordinated manner. Team members 
felt that it was the responsibility of Public Safety Canada to define 
protocols and formalize agreements among members. At the time of 
our audit, these issues had not been resolved. Public Safety Canada 
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officials told us that the role it could play in this type of emergency is 
unclear, as the three departments on the federal CBRNE team have 
the expertise, resources, and responsibility, while Public Safety Canada 
has none of these. 


744 Recommendation. As stipulated in the Emergency Management 
Act, Public Safety Canada should ensure that its coordination role for 
the federal response to an emergency is well-defined and that the 
operational policies and plans that departments will follow are updated 
and consistent. 


The Department’s response. Agreed. Public Safety Canada will 
maintain the Federal Emergency Response Plan and its components as 
an evergreen document. This includes ensuring the development of 
policies and event-specific plans that outline operational protocols and 
departmental roles and responsibilities, and reviewing these plans to 
ensure a coordinated approach as necessary. 


Standards to promote interoperability are still under development 


7.45 The 2004 National Security Policy called for equipment and 
communications to be interoperable or compatible so that first 
responders could work together better. In response to our 2005 audit 
chapter, Public Safety Canada agreed to collaborate with a research 
group to develop standards for equipment for use in chemical, 
biological, radiological, or nuclear emergencies. The equipment is used 
in a variety of emergency response situations, and it includes fire and 
heavy urban search and rescue vehicles, personal suits, gear worn by 
first responders to protect against hazardous materials, and 
communications systems. 


7.46 First responders have identified voice communications as the 
main constraint to their interoperability. Capability gaps remain in 
communications interoperability that limit the ability of fire, police, 
and ambulance services to talk to one another and to communicate 
across jurisdictions during an emergency. Public Safety Canada officials 
told us that its role is not to establish standards but to assist first 
responder groups that purchase and use the equipment to develop 
their own standards. Public Safety Canada completed a draft 
document on a national approach for communications interoperability 
but has yet to present the draft to provincial officials for approval. For 
other types of equipment, Public Safety Canada is currently assisting 
groups to establish standards for personal protective equipment. 
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7.471 As noted in our 2005 audit, while the federal government could 
use directed funding to promote standardized equipment, officials told 
us that it has not done more due to a lack of resources. About 

$5 million in federal funding is available through an existing 
cost-shared program. Under this program, choices of equipment 
purchases are left to the provinces. 


A strategy for protecting critical infrastructure has been slow to develop 


7.48 Public Safety Canada is the lead federal department for 
coordinating the protection of Canada’s critical infrastructure. Critical 
infrastructure consists of those physical and information technology 
facilities, networks, services, and assets that, if disrupted or destroyed, 
would have a serious impact on the health, safety, and security or 
economic well-being of Canadians or the effective functioning of 
governments in Canada. The Emergency Management Act stipulates 
that the Minister of Public Safety is to provide advice and to analyze 
and evaluate federal departmental emergency management plans, 
which include critical infrastructure plans. 


7.49 We examined whether Public Safety Canada was providing a 
leadership role in developing and implementing a national strategy for 
critical infrastructure protection. Namely, we examined its initiatives 
to provide advice and promote standards to other federal and 
provincial or territorial authorities. 


7.50 In February 2001, the federal government identified the need to 
provide national leadership to protect Canada’s critical infrastructure 
from the risks of failure or disruption. Following the terrorist attacks in 
the United States on September 11, 2001, the federal government 
allocated $190 million over five years to improve critical infrastructure 
protection and emergency management capacity across the federal 
government. In 2004, the National Security Policy directed the federal 
departments to work with provinces, territories, and the private sector 
on initiatives to improve national capabilities to protect critical 
infrastructure. In its 2004-05 Report on Plans and Priorities, Public 
Safety Canada committed to the development and release of a 
National Strategy for Critical Infrastructure by spring 2005. 


7.51 While not meeting its target date in 2005, Public Safety Canada 
started to work with provinces, territories, and the private sector to 
develop a plan to implement a proposed National Strategy for Critical 
Infrastructure. In the strategy, 10 key sectors involved in critical 
infrastructure were identified, and federal departments were 
designated to head each sector. We found that Public Safety Canada 
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has consulted with representatives of government and private sector 
organizations in order to draft the National Strategy for Critical 
Infrastructure. It expects that implementation will take three years 
once the strategy is formally approved. Department officials told us 
that they are continuing to work on implementation while they await 
formal approval. From its monitoring, Public Safety Canada found that 
progress was more advanced in some sectors than in others toward 
completion of the 37 milestones necessary for their emergency 
management to be fully operational (Exhibit 7.3). 


Exhibit 7.3 The 10 critical infrastructure sectors show varied progress toward their emergency 
management being operational 


Sectors 


Government 

Safety 

Manufacturing 

Transportation 

Health 

Water 

Food 

Finance 

Information and Communication Technology 


Energy and Utilities 


0 5 10 15 20 37 
Number of completed milestones, toward a total of 37 


Source: Based on data provided by Public Safety Canada 


7.52 At the time of our audit, Public Safety Canada had started to 
develop guidance to promote a consistent approach to critical 
infrastructure risk assessments and protection efforts. However, this 
guidance had not been finalized or distributed to departments 
designated to head the sectors. We found that, in the absence of 
guidance from Public Safety Canada, departments have been 
developing their own approaches, without the assurance that they will 
result in plans that are coordinated and consistent across government. 


Canada’s critical infrastructure remains undetermined 


7.53 With the proposed critical infrastructure strategy, Public Safety 
Canada has taken the first step toward getting a complete picture of 
the infrastructure considered important at the federal, provincial, and 
municipal levels. However, to get this picture requires input from many 
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different partners in government and the private sector. At the time of 
our audit, the critical infrastructure that needs to be protected had not 
yet been determined. Public Safety Canada had begun to map the 
infrastructure of 14 major Canadian cities. However, none of this 
information had been validated for its significance at the federal level. 
While certain assets may be‘deemed critical to an industry, 
municipality, province, or territory, those assets may not be critical at 
the federal or national level. This information is key for industry and 
all levels of government to allocate resources and develop their own 
protection plans. 


7.54 There have been challenges to progress, specifically 


¢ the determination of which critical infrastructure needs 
protection; 


¢ the determination of what resources are available to protect 
critical infrastructure at the federal, provincial, and territorial 
levels and of where weaknesses or gaps exist; and 


¢ hesitation on the part of private sector owners and operators of 
some infrastructure to share information that would identify 
potential vulnerabilities that could provide competitors with an 
advantage. 


7.55 While Public Safety Canada can move forward to develop 
policies and programs without resolving these issues, the unresolved 
issues will remain an impediment to achieving full success if they are 
not addressed. The proposed national strategy includes information 
sharing and protection as a key strategic objective, and the Access to 
Information Act has been amended to protect critical infrastructure 
information supplied by third parties. However, department officials at 
Public Safety Canada told us that, while they can provide advice and 
coordination to departments, it is the responsibility of operational 
departments to identify Canada’s critical infrastructure and determine 
how it should be protected before a national, coordinated approach 
can be implemented. 


7.56 Public Safety Canada has provided no guidance to departments 
to ensure that they determine what critical infrastructure needs to be 
protected. Furthermore, there is little guidance to departments 
responsible for sectors to determine what assets or facilities are critical 
to the federal government. This information is essential for a 
coordinated approach to critical infrastructure protection. 
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The energy and utilities sector is making progress on protecting critical 
infrastructure 


7.57 We examined the energy and utilities sector in more detail as it 
was seen to have made considerable progress in efforts to identify and 
protect critical infrastructure. Led by Natural Resources Canada, the 
sector is organized and has regular meetings and classified briefings to 
industry and government officials. 


7.58 Natural Resources Canada is in the process of adding 
infrastructure information to maps, including not only pipelines and 
transmission lines, but also railways, telecommunications, and strategic 
buildings and structures. While these efforts are expected to 
complement the proposed national strategy, we note that this project 
was developed separately from Public Safety Canada, which has since 
initiated its own separate mapping of critical infrastructure. 


Cyber security has recently received more attention, but significant challenges 
remain 


7.59 Threats to computer-based critical infrastructure, including 
federal information systems, are evolving and growing. In April 2009, 
the Minister of Public Safety stated that there have been repeated 
attacks against this country’s computer systems. These cyber attacks 
may be initiated by individuals or groups and may be unintentional, 
amateur, or foreign state-sponsored espionage and information warfare, 
and present an ever-changing and evolving threat. Cyber attacks could 
have very damaging consequences. For example, computer and 
communications networks are used to control such things as our 
electrical grid, with varying vulnerabilities. Recently, the United States 
and the United Kingdom have significantly increased their efforts to 
fight cyber threats. 


7.60 Public Safety Canada is in the process of developing a cyber 
security strategy—a commitment first made in the 2004 National 
Security Policy. While it has been working on a draft strategy, at the 
time of our audit, it had no date scheduled for its formal approval. 
Although the commitment was made in 2004, progress has been slow 
until this past year. Public Safety Canada has identified the key 
elements of a cyber strategy and has initiated action on a list of current 
cyber security initiatives along with other federal government 


departments. 


7.61 Recommendation. Based on the responsibilities outlined in the 
Emergency Management Act, Public Safety Canada should provide 
policies and guidance for departmental sector heads to determine their 
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infrastructure and assess its criticality, based on risk and its significance 
to the safety and security of Canadians; it should establish policies and 
programs to prepare plans to protect the infrastructure. 


The Department’s response. Agreed. Based on the responsibilities 
outlined in the Emergency Management Act, Public Safety Canada will 
provide tools and guidance for sectors to determine their processes, 
systems, facilities, technologies, networks, assets, and services. Public 
Safety Canada will also provide tools and guidance for departmental 
sector heads to assess the infrastructure’s criticality based on risks and 
its significance to the safety and security of Canadians, and will 
establish policies and programs to prepare plans for their protection. 


Conclusion 


7.62 We found that Public Safety Canada has not exercised the 
leadership necessary to coordinate emergency management activities, 
including protection of critical infrastructure in Canada. While it has a 
challenging role, Public Safety Canada still needs to develop the 
policies and programs that would help clarify its leadership and 
coordination role for the emergency management activities of 
operational departments. Public Safety Canada has taken the first step 
by developing the interim Federal Emergency Response Plan. In our 
opinion, to make further progress, the plan would benefit from 

formal government approval and a better definition of roles and 
responsibilities of all players, as well as the capabilities needed for 

an integrated, coordinated approach to emergency response. 


7.63 Public Safety Canada has drafted a strategy to protect Canada’s 
critical infrastructure, but it has not been formally approved. However, 
at the time of our audit, the critical infrastructure that needs to be 
protected had not yet been determined. It has categorized critical 
infrastructure into 10 sectors, each headed by a federal department. 


7.64 We found that Public Safety Canada had made slow progress 
until this past year on its 2004 commitment to develop a cyber security 
strategy, although threats to computer-based critical infrastructure, 
including federal information systems, are evolving and growing. While 
it has been working on a draft strategy, at the time of our audit, it had 
no date scheduled for its formal approval. Public Safety Canada has 
identified the key elements of a cyber security strategy and has 
initiated action on a list of current cyber security initiatives. along 
with other federal departments and agencies. 
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7.65 Over the period of our audit, Public Safety Canada, along with 
other federal departments and agencies, had made limited progress in 
enhancing the response to and recovery from emergencies in a 
coordinated manner. However, their rate of progress has improved, 
especially in the past year. Public Safety Canada has established a 
Government Operations Centre, which is connected to other federal 
departments and agencies. The centre has enabled Public Safety 
(Canada to make considerable progress in coordinating response 
activities in times of crisis, as it keeps other departments informed of 
the status of events on a real-time basis. It also produces regular 
situation awareness reports for such issues as the HIN1 virus, which 
allow decisions to be based on a common set of facts. However, 
improvements can be made in identifying and implementing lessons 
learned from real emergencies and exercises. In its responsibility as the 
lead federal department for emergency management policies and plans, 
including chemical, biological, radiological, nuclear, and explosives, 
Public Safety Canada has not clarified the decision-making processes 
and operational protocols for emergency response activities. 


7.66 Public Safety Canada is making progress in promoting standards 
for personal protective equipment used in responding to emergencies. 
However, key interoperability gaps remain for voice communications, 
limiting the ability of various fire, police, and ambulance services to 
work together in an emergency. The Department has directed little or 
no funding toward standardizing equipment. 
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About the Audit 


All of the audit work in this chapter was conducted in accordance with the standards for assurance - 
engagements set by The Canadian Institute of Chartered Accountants. While the Office adopts these 
standards as the minimum requirement for our audits, we also draw upon the standards and practices of 
other disciplines. 


Objectives 
The objectives of this audit were to 


« determine whether Public Safety Canada can demonstrate that it has exercised leadership by 
coordinating emergency management activities, including critical infrastructure protection in Canada; 
and 


« determine whether Public Safety Canada, along with federal departments and agencies, can 
demonstrate progress in enhancing the response to and recovery from emergencies in a coordinated 
mannet. 


Scope and approach 


This audit examined federal efforts to improve the nation’s readiness and resiliency to respond to incidents 
or attacks, through improved coordination of emergency management activities at the federal level, and 
through work with provinces and territories to achieve unified and integrated response and recovery 
operations. While the focus of the audit was Public Safety Canada, audit work was also conducted at the 
Privy Council Office, National Defence, the Royal Canadian Mounted Police, the Public Health Agency 
of Canada, Health Canada, Natural Resources Canada, the Canadian Food Inspection Agency, and the 
Canada Border Services Agency. 


We followed up on selected recommendations made in our April 2005 chapter, National Security in 
Canada, regarding emergency preparedness, including response capabilities for a chemical, biological, 
radiological, or nuclear event. Public Safety Canada has the lead responsibility for addressing the majority 
of these recommendations. We also followed up on selected recommendations from the June 2005 House 
of Commons Standing Committee on Public Accounts report that supported our audit chapter with 
several recommendations to federal departments. 


The audit did not examine emergency management activities of the provinces and territories; it focused on 
Public Safety Canada’s coordination of emergency management among federal departments along with the 
provinces and territories. The audit did not examine the security activities carried out in preparation for 
the 2010 Olympic and Paralympic Games as responsibility for these activities was assigned to the Office of 
the Coordinator for 2010 Olympics and G8 Security reporting to the National Security Advisor. 
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Criteria 


Listed below are the criteria that were used to conduct this audit and their sources. 


We expected that Public Safety Canada would exercise 
leadership by coordinating federal emergency management 
activities, as described in legislation and policies. 


Department of Public Safety and Emergency Preparedness 
Act 


Emergency Management Act, sections 3 and 4 


National Security Policy (2004), page 22 


We expected that Public Safety Canada would coordinate federal 
emergency management activities with those of the provinces 
and territories to provide timely and coordinated support to 
communities in an emergency. ; 


Emergency Management Act, section 4.1(f) 


National Security Policy (2004), page 25 


We expected that Public Safety Canada would regularly test and 
exercise federal emergency management plans. 


Emergency Management Act, section 4.1(a) 
National Security Policy (2004), page 27 


National Security in Canada: Report of the Standing 
Committee on Public Accounts, June 2005, page 10 


We expected that Public Safety Canada would have a risk-based 
plan to lead and coordinate critical infrastructure protection 
efforts, and to reduce vulnerability to cyber attacks and 
accidents, by 


¢ adopting an all-hazards approach 


* agreeing upon roles and responsibilities for the federal 
government and others 


¢ determining what critical infrastructure should be protected 

* assessing the threats and risks to these assets 

* prioritizing risks and resources to protect critical infrastructure 
* implementing protective programs 


* developing measures to monitor and assess effectiveness 


National Security Policy (2004), page 26 


Public Safety Canada, Securing an Open Society: One Year 
Later (2005), page 23 


Emergency Management Act, sections 3 and 4 


We expected that Public Safety Canada and selected federal 
entities would use a risk-based approach to identify the 
resources needed and to coordinate the response to and 
recovery from emergencies. . 


Emergency Management Act, sections 4 and 6 


Treasury Board of Canada Secretariat, Management 
Accountability Framework, Round V—Risk Management, 
sections 9.1 to 9.4 


Treasury Board of Canada Secretariat, Integrated Risk 
Management Framework 
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Criteria : east foe Sources 


We expected that Public Safety Canada would promote a e Emergency Management Act, section 4 
common approach to emergency management, including the 


adoption of standards and best practices. = HNatioltal Se cunty ONG a 200+) pase 20 


We expected that Public Safety Canada, together with its federal e Emergency Management Act, section 4(1)(n) 
partners, would provide emergency management training, based 


on a needs assessment and risk-based plan. ¢ Government of Canada, The Budget Plan 2001, page 100 


Management reviewed and accepted the suitability of the criteria used in the audit. 


Period covered by the audit 


This audit covers the performance of federal departments and agencies and events taking place since our 
last audit of this subject reported in April 2005. 


Audit work for this chapter was substantially completed on 15 June 2009. 


Audit team 


Assistant Auditors General: Wendy Loschiuk and Hugh McRoberts 


Principal: Gordon Stock 
Director: Carol McCalla 


Jenna Lindley 

Sean MacLennan 
Steven Mariani 
John-Patrick Moore 
Nicolette O’Connor 
Bridget O’Grady 
Stacey Wowchuk 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 7. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


Establishing policies and programs 


7.26 The Privy Council Office and 
Public Safety Canada should ensure 
that all components of the Federal 
Emergency Response Plan are 
completed and should obtain 
government approval for the plan. 


(7.17-7.25) 


7.31 As stipulated in the Emergency 
Management Act, Public Safety Canada 
should establish policies and programs 
and provide advice for departments to 
follow when identifying risks and 
developing their emergency 
management plans. (7.27—7.30) 


The Privy Council Office and the Department’s response. 
Agreed. The Privy Council Office and Public Safety Canada will 
seek approval for the completed Federal Emergency Response 
Plan (FERP) at the earliest possible date and the supporting 
Emergency Support Functions (ESFs) prior to the end of 

the 2009-10 fiscal year. Public Safety Canada will seek approval 
of the National Emergency Response System (NERS), an annex 
to the FERP which articulates how the FERP supports provincial 
and territorial emergency response plans, by the end 

of August 2010. Public Safety Canada will organize information 
sessions with departmental executive committees to brief 
departments on the FERP and their associated roles and 
responsibilities. The FERP and its components will be 
maintained as an evergreen document. 


The Department’s response. Agreed. In keeping with the 
all-hazards approach to emergency management, Public Safety 
Canada is leading the development of an Emergency 
Management Planning Framework that will provide departments 
and agencies with guidance, tools, and best practices for 
developing emergency management plans. It is also working with 
federal departments to develop an all-hazards risk assessment 
framework. Under the Emergency Management Act, it is the 
responsibility of each minister accountable to Parliament for a 
government institution to identify the risks that are within or 
related to his or her area of responsibility. 
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7.32 As stipulated in the Emergency 
Management Act, Public Safety Canada 
should analyze and evaluate the 
emergency management plans prepared 
by departments to ensure that they are 
prepared according to the policies, 
programs, and advice provided, and it 
should identify potential gaps or risks to 
a coordinated emergency management 


response. (7.27—7.30) 


The Department’s response. Agreed. Public Safety Canada is 
developing the Emergency Management Planning Framework, 
which will include performance measurements that will allow 
Public Safety Canada to analyze and evaluate emergency 
management plans produced by departments and agencies. The 
Framework will also include self-assessment tools for 
departments and agencies. Public Safety Canada is currently 
developing an approach to implement this initiative. - 


Coordinating federal emergency management 


7.44 As stipulated in the Emergency 
Management Act, Public Safety Canada 
should ensure that its coordination role 
for the federal response to an 
emergency is well-defined and that the 
operational policies and plans that 
departments will follow are updated and 
consistent. (7.33—7.43) 


The Department’s response. Agreed. Public Safety Canada will 
maintain the Federal Emergency Response Plan and its 
components as an evergreen document. This includes ensuring 
the development of policies and event-specific plans that outline 
operational protocols and departmental roles and 
responsibilities, and reviewing these plans to ensure a 
coordinated approach as necessary. 


Protecting critical infrastructure 


7.61. Based on the responsibilities 
outlined in the Emergency Management 
Act, Public Safety Canada should 
provide policies and guidance for 
departmental sector heads to determine 
their infrastructure and assess its 
criticality, based on risk and its 
significance to the safety and security of 
Canadians; it should establish policies 
and programs to prepare plans to 
protect the infrastructure. (7.48-7.60) 


The Department’s response. Agreed. Based on the 
responsibilities outlined in the Emergency Management Act, 
Public Safety Canada will provide tools and guidance for sectors 
to determine their processes, systems, facilities, technologies, 
networks, assets, and services. Public Safety Canada will also 
provide tools and guidance for departmental sector heads to 
assess the infrastructure’s criticality based on risks and its 
significance to the safety and security of Canadians, and will 
establish policies and programs to prepare plans for their 
protection. 
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What we examined 


Why it’s important 


Strengthening Aid Effectiveness— 
Canadian International Development 
Agency 


Main Points 


The Canadian International Development Agency (CIDA) 
administers the bulk of Canada’s official development assistance 
program. Its mission is to lead Canada’s international effort to help 
people living in poverty. It does not have governing legislation that 
defines its mandate and role. 


In its 2002 Policy Statement on Strengthening Aid Effectiveness, 
CIDA committed to align its efforts with recipient countries’ needs 
and priorities; harmonize its activities with those of other donors; and 
use new forms of aid known as program-based approaches, in which 
donors coordinate support to the budgets of recipient governments or 
local organizations for a development program delivered using local 
systems and procedures. Recognizing that its programming is widely 
dispersed, CIDA has also committed to focus its aid in fewer sectors in 
order to make a more meaningful contribution. 


We examined how the Agency is implementing these commitments. 
We looked at management processes in place at its head office to guide 
and sustain the implementation of the commitments. In addition, we 
selected five countries and looked at how CIDA plans its activities 
there, selects projects for funding, and monitors project risks. 


Audit work for this chapter was substantially completed 15 May 2009. 


Development assistance is a key means to help developing countries 
reduce poverty. It is estimated that over 1.4 billion people are living on 
less than US$1.25 a day—the latest international poverty line set by 
the World Bank. CIDA’s commitments reflect principles of aid 
effectiveness that international donor countries have recognized will 
lead to more sustainable and self-reliant development. 


By its nature, international development requires a long-term effort 
and stable, predictable programming. CIDA has declared aid 
effectiveness to be its overarching priority, recognizing that this 
involves risks of a different nature and level and significantly changes 
the way it needs to work with other donors and recipient governments. 
Robust management processes—which include specific action plans, 
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Z Chapter 8 


What we found 


performance targets, assigned accountabilities, and management 
review—are crucial to sustain implementation of its commitments and 
directions over time. 


¢ In the field, CIDA is working in collaboration with other donors to 
align its development assistance with the needs of the recipient 
countries we looked at. It is working to harmonize its efforts with 
those of other donors and to use program-based approaches where 
appropriate. Its selection of projects for funding is based on a 
rationale consistent with principles of aid effectiveness. We were told 
that CIDA is well regarded in the field and that its partners 
appreciate its efforts. However, the complex and lengthy processes 
required to obtain corporate approval for project funding have long 
been criticized within and outside the Agency. 


* Corporate support in the Agency for program-based approaches has 
been neither uniform nor timely. In the early years after the 2002 

Policy Statement was released, management provided little specific 
direction to field operations on how and when to use program-based 
approaches, and there are no quantitative targets for the extent of 
their use. The Agency’s internal processes do not clearly stipulate 
the conditions under which it would or would not participate in a 
program-based approach or the types of risk assessment to be done 
before agreeing to participate. However, it had undertaken some 
form of risk assessment for the program-based approaches we 
examined or it had relied on its donor partners’ analyses. It also 
monitored project risks. 


« CIDA’s main planning documents—Country Development 
Programming Frameworks—for the countries we examined have 
expired, and the process used to prepare and maintain them has been 
discontinued. The Agency is developing a new planning process but, 
by the end of our audit, the status of the new plans was unclear 
because none had received formal approval or been communicated 
broadly within CIDA. None have been made public. This means that 
donors, recipient governments, and program staff are unclear about 
the Agency’s direction and long-term commitment to specific 
countries or regions. 


- As a result of lack of clear direction and action plans, coupled with 
broadly defined and shifting priority sectors, CIDA is not realizing its 
goal of making a more meaningful Canadian contribution in a 
country or region by focusing its aid more narrowly. It has made 
limited progress in determining its strengths relative to other donors 
as well as the type of expertise it requires to support its priority 
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sectors and how best to provide it. The lack of direction and shifting 
priorities have undermined the long-term predictability of the 
Agency’s efforts. 


- Many of the weaknesses we identified can be traced to a lack of 
corporate management processes to guide and monitor the 
implementation of CIDA’s aid effectiveness commitments. The 
Agency’s early intentions were not matched with specific action 
plans and follow-through, and it has yet to develop a comprehensive 
strategy for implementing its commitments. In addition, frequent 
changes in policy direction and substantial turnover of senior 
personnel in recent years have posed significant challenges for 
CIDA’s aid effectiveness agenda. 


The Agency has responded. The Agency agrees with all of our 
recommendations. Its detailed responses follow the recommendations 
throughout the chapter. 
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Millennium Development Goals 

1. Eradicate extreme poverty and hunger 

2. Achieve universal primary education 

3. Promote gender equality and empower women 
4. Reduce child mortality 

5. Improve maternal health 

6. Combat HIV/AIDS, malaria, and other diseases 
7. Ensure environmental sustainability 


8. Develop a Global Partnership for Development 
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Introduction 


Canada’s aid agency 


8.1 The Canadian International Development Agency (CIDA) is 
Canada’s aid agency. Its mission is to lead Canada’s international 
efforts to help people living in poverty. Poverty remains a rampant 
problem throughout the world. An estimated 1.4 billion people are 
surviving on less than US$1.25 a day—the latest international poverty 


line set by the World Bank. 


8.2. Recognition of such poverty provided the backdrop to one of the 
largest gathering of world leaders in human history—in September 2000, 
the Millennium Summit of the United Nations in New York. At this 
summit, Canada and 188 other nations adopted the United Nations 
Millennium Declaration and the Millennium Development Goals— 
eight goals that serve as the yardstick for measuring improvements in 
the most critical areas of human development by 2015. 


8.3. Development assistance, in its many forms, is one of the 


~ means that aid donors use to help developing countries achieve 


the Millennium Development Goals. The Development Assistance 
Committee (DAC) of the Organisation for Economic Co-operation 
and Development (OECD) reports that official development 
assistance disbursements by member countries (which includes 
Canada) rose to over US$120 billion in 2008, from around 

US$58 billion in 2002. Canada ranks 9th in terms of total 
disbursements and 16th in terms of percentage of the gross national 
income among the 22 member countries. In 2008, Canada disbursed 
a total of US$4.7 billion in official development assistance. 


8.4 Although several federal departments (such as the Department 
of Finance Canada and Foreign Affairs and International Trade 
Canada) provide official development assistance, CIDA administers 
the bulk of Canada’s contributions. The Agency’s budget had grown to 
about CDN$3 billion in the 2007-08 fiscal year, from about 

CDN$2 billion in the 2002-03 fiscal year. 


8.5 CIDA reports to Parliament through the Minister of 
International Cooperation. Its staff of approximately 1,800 (full-time 
equivalents) includes Canadians who work both at headquarters (the 
vast majority) or in Canadian missions abroad. Country desks within 
CIDA are responsible for specific country-to-country programming. 
They are headed by a country director and typically consist of staff 
located at headquarters and at missions abroad. Field staff are led by a 
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head of aid. Country desks are supported by locally engaged personnel 
and technical experts on contract, including technical advisers 
operating within support units in the field. These contractors number 
in the thousands. 


8.6 Three main branches in CIDA provide development assistance. 
Each branch has a separate internal mandate: 


¢ The Multilateral and Global Programs Branch provides core 
contributions to multilateral development institutions and to 
targeted regional and global programs. It also manages 
humanitarian assistance. 


¢ The Canadian Partnership Branch supports cost-sharing 
partnerships with civil society and private-sector organizations for 
development aid. 


¢ The Geographic Programs Branch oversees all country-to-country 
aid relationships. 


8.7 Currently, CIDA’s geographic (country-to-country) programming 
takes place in more than 60 countries. The Agency has recognized that 
it disperses its aid more widely than most other aid agencies, and, at the 
turn of the millennium, it committed to focusing its efforts on a limited 
number of the world’s developing countries. On 23 February 2009, the 
Agency formally announced—with Cabinet approval—its intention to 
focus its aid on 20 countries. Exhibit 8.1 illustrates CIDA’s current 
organizational structure, lists the 20 countries of focus, and outlines 
spending for the 2008-09 fiscal year. 


8.8 The Agency does not usually deliver aid itself. Instead, it funds 
other organizations, such as Canadian “executing agencies” (volunteer 
and non-government aid organizations, consulting companies, and 
universities), and various international and United Nations’ 
organizations; in some cases, it funds the governments of recipient 
countries directly. 


8.9 The Agency is considered to be a federal government 
department under the Financial Administration Act. The financial 
delegation of authorities governs project approvals. The Minister of 
International Cooperation has the authority to approve projects of up 
to $20 million, while the Treasury Board must approve projects of more 
than $20 million. CIDA’s president and vice-presidents have the 
authority to approve projects of up to $5 million. Country directors can 
approve projects of up to $500,000. 
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Exhibit 8.1 The Canadian International Development Agency’s organizational chart and 2008-09 fiscal year spending ($ millions) (unaudited) 


Geographic programs 


$1,784 


Minister 
of International 
Cooperation 


Aid expenditure to 
countries and regions of focus _ 


Aid expenditure to other countries 
and regions and other expenditures** 


Multilateral and 
global programs 


President $575 


SL S00 Rae er Gh i Nerne: By bree ers ee sree 
Afghanistan 220 


$3,583 


Bangladesh 06 
Canadian partnership Bolivia 4 
$260 Caribbean 37 
Colombia 6 

Other branches* — - 
$159 Haiti 122 
Honduras 13 

Indonesia 70 

Mali 90 

Mozambique 69 

Pakistan 33 

Peru ii 

Senegal 67 

Sudan 45 

Tanzania. 84 

Ukraine 18 

Vietnam 29 


West Bank/Gaza 


Note *: Includes Human Resources, Chief Financial Officer, Chief Information Officer, Communications and Strategic Policy & Performance 
Note **: Other expenditures include operating expenditures and Canada Funds for Local Initiatives 


Source: The Canadian International Development Agency (CIDA) 


8.10 CIDA does not have comprehensive governing legislation that 
defines its mandate and role. A combination of influences affects the 
Agency’s programming, including how and how much to spend on 
development assistance, where to concentrate its efforts, what the goal 
of those efforts is, and what means it uses to achieve its goals. These 
influences include Government of Canada policy objectives, the 
priorities of the Minister of International Cooperation, and CIDA’s 
own operational policies and strategies. 


8.11 Since 2000, five different ministers of International Cooperation 
and four different Agency presidents have led CIDA. The Agency has 
had to adjust its programming in response to frequent and significant 
changes in government and ministerial policy and direction. These 
changes have included Canada’s overall development objectives, the 
countries and regions CIDA focuses on, and the sectors and global 
issues Canada and CIDA have considered priorities. Funding 
allocations have also changed to reflect these shifts. The significant 
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In contrast to the traditional approach to aid, 
which is characterized by individual stand-alone 
projects, a program-based approach is based 
on coordinated support from donors for a 
development program of the recipient 
government or of a local organization. This 
approach Is intended to result in more 
sustainable and self-reliant development and 
includes four main elements: 


leadership by the host country or 
organization; 


a single program and budget framework: 


donor coordination and harmonization of 
procedures; and 


efforts to increase the use of local procedures 
over time, with regard to program design and 
implementation, financial management, and 

monitoring and evaluation. 


turnover of the Agency’s senior management has posed an ongoing 
additional challenge to providing a stable operating environment. 


8.12 CIDA does not act alone. It is part of a larger development 
community, globally and at a country-specific level—a community that 
also influences CIDA’s programming. This community includes aid 
agencies from other donor countries, development banks, United 
Nations and other multilateral institutions, civil society and non- 
government organizations, and the governments of recipient countries. 


The Agency and aid effectiveness 


8.13 Theories about what makes aid effective have evolved over time. 
A turning point was the 1996 publication by the OECD Development 
Assistance Committee, Shaping the 21st Century: The Contribution of 
Development Cooperation, which outlined a series of key principles for 
effective development, such as basing aid on the recipient country’s 
priorities and improving coordination among individual donors. 
International experience has also given rise to new ways of designing and 
funding aid, now commonly referred to as program-based approaches. 


8.14 Against this backdrop, CIDA released its 2002 Policy Statement 
on Strengthening Aid Effectiveness, which identified a broad 
framework for geographic and sectoral allocation of the Agency’s 
resources, changes in the use of tied aid, and steps to be taken to 
strengthen CIDA as an institution. Of particular importance for this 
audit, the policy commits the Agency to harmonize its aid with other 
donors, to align its efforts with recipient countries’ development 
strategies, and to use program-based approaches. Elsewhere, in 
addition to these internationally recognized principles of aid 
effectiveness, CIDA has committed to focus its aid on particular 
sectors, recognizing that while the needs of developing countries are 
unlimited, its resources are not. Examples of sectors include education, 
health, agriculture and democratic governance. CIDA has stated and 
re-stated its commitments and other aspects of its aid effectiveness 
agenda in the successive reports on plans and priorities it has 
submitted to Parliament since 2001. 


8.15 CIDA continues to be influenced by international developments.. 
Through a series of international high-level meetings held in Rome, 
Italy (2003), Marrakech, Morocco (2004), and Paris, France (2005); 
Canada, recipient and donor countries, and international institutions 
have further elaborated and agreed on a set of principles, specific 
actions, and targets to guide the management of aid, articulated in 

the 2005 Paris Declaration on Aid Effectiveness. 
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8.16 While it is commonly acknowledged that the Paris Declaration 
is gradually taking hold in recipient countries; most recently, at a 
high-level meeting in Accra, Ghana (2008), Canada and other donors 
committed to a number of concrete steps to accelerate progress with 
respect to aid effectiveness. In recognition of the long-term nature of 
development assistance, they also committed to providing more 
predictable and transparent aid that would flow over multiple years. 


Focus of the audit 


8.17 This audit examined how CIDA has put into practice its aid 
effectiveness commitments concerning alignment, harmonization, 
program-based approaches, and focusing of its resources. We looked at 
three separate but related functions in the organization: planning and 
support of country programs, project selection and monitoring, and 
corporate management processes and oversight. 


8.18 At the country level, we examined how CIDA develops its 
long-range country plans, including whether they reflect its aid 
effectiveness commitments. We concentrated on the Agency’s 
Geographic Programs Branch and chose for detailed examination 
five countries on its list of countries of concentration: Bangladesh, 
Ethiopia, Ghana, Honduras, and Vietnam. We examined five projects 
from each country to determine whether the selection of projects was 
based on a sound rationale and whether CIDA was assessing and 
monitoring risks as it implemented them. At the corporate level, we 
examined whether the Agency has put in place management processes 
to direct and sustain implementation of its commitments over time. 


819 More details on the audit objectives, scope, approach, 
and criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


8.20 Conditions in recipient countries have a significant impact 

on the Canadian International Development Agency’s (CIDA’s) 
programming. The Agency must take into account the unique social 
conditions of the people; the economic, legal, and political systems of 
each government; and the presence and activity of other donors and 
institutions when planning its work. Careful planning that reflects 
such conditions increases the likelihood that aid will be successful and 
its benefits long-lasting. Exhibit 8.2 includes selected statistics on each 
of the countries we selected for review as well as some indicators of 
CIDA’s and Canada’s development aid involvement there. 
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8.21 Until recently, the Agency used country development 
programming frameworks to guide the selection of individual projects 
(which, when combined, make up the country program) and the 
spending of millions of dollars of aid over a five-year time frame. In 
December 2003, CIDA updated its guide for these frameworks, 
requiring them to reflect the major elements of its 2002 Policy 
Statement on Strengthening Aid Effectiveness and to be based on an 
analysis of the plans and priorities of recipient governments, the 
activities of other donors, and on the Agency’s strengths relative to 
other donors.. The frameworks were meant to be updated if, among 
other things, there were significant political or economic changes in a 
recipient country. 


8.22 We examined the way CIDA plans and supports its work 
(including the use of formal frameworks), specifically how it aligns its 
aid with the needs of recipient countries, how it coordinates and 
harmonizes its work with other donors, how it directs the use of 
program-based approaches, and how it pursues greater focus of its 
efforts. We examined the frameworks for Bangladesh, Ethiopia, and 
Vietnam (which covered programming years subsequent to the release 
of the 2002 Policy Statement and were prepared according to the 2003 
updated guide) as well as for Ghana and Honduras (which were 
already in effect at the time the 2002 Policy Statement was released). 


Geographic aid is aligning with recipient country needs 


8.23. The principle of alignment is largely based on the premise that 
aid works better and its benefits last longer if donors provide it where 
developing countries need and want it. Alignment means that donors 
should be responsive, not prescriptive, in meeting a country’s needs. 


8.24. Developing countries usually express their needs in a poverty 
reduction strategy paper. These strategy papers typically 


* define the country’s development needs; 


¢ describe its macroeconomic, structural, and social policies and 
programs; and 


¢ identify the external financing needs and possible sources a 
government will pursue to promote economic growth and reduce 


poverty. 


We assessed how CIDA’s country development programming 
frameworks incorporated the recipient countries’ needs and priorities 
as expressed in their strategy papers. 
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8.25 We found that CIDA’s programming frameworks considered and 
reflected recipient countries’ needs at the time they were written; and 
they demonstrated high-level alignment, bearing in mind that the 
strategy papers tend to identify a vast range of needs. That said, 
conditions and priorities in recipient countries change over time, 
posing challenges for CIDA in its attempt to align with a country’s 
priorities. While Bangladesh, Ethiopia, Ghana, and Vietnam have 
poverty strategy papers in effect, Honduras’ paper effectively expired 
in 2007 and the government has not yet replaced it. 


8.26 Therefore, pursuing alignment on an ongoing basis is important, 
and maintaining current and detailed knowledge of the country context 
is critical. CIDA personnel and locally engaged experts we interviewed 
had knowledge of each country’s context and priorities. They had 
established formal ways of exchanging information with recipient 
governments (often through joint government-donor committees), 

as well as informal networks to keep their knowledge current. 


Efforts are being made to harmonize with other donors 


8.27 Many donors are present in developing countries, leading to 
potential fragmentation of their efforts and placing a heavy 
administrative burden on the government, which must cope with the 
rules and expectations of multiple agencies. By sharing information 
and working together, harmonization can lead to simplified procedures 
and joint arrangements for designing, managing, and delivering aid, 
including the use of program-based approaches. Not only can this 
reduce the administrative burden for recipient countries, it can also 
increase the benefits to those countries, through donors’ combined and 
complementary efforts. 


8.28 In order to intensify its efforts toward donor coordination, 
CIDA released an action plan to promote harmonization in 2004. 
The document established a range of initiatives designed to respond 
to challenges and needs that existed at the time. It also included a | 
number of indicators for measuring its performance and assigned 
responsibility to several of its branches for taking action. However, 
over time, the Agency faltered in implementing the plan. We could not 
locate the monitoring and progress reports that it called for, so the 
Agency could not show its performance against planned targets and 
indicators. The executive committee has not followed through on an 
internal proposal to renew the action plan. 


8.29 The country development programming frameworks for the 
countries we examined mentioned the goal of improved harmonization. 
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They included brief descriptions of other donors’ activities (such as 
levels of funding and sectors of programming) but lacked in-depth 
analysis of individual donors’ strengths. Without that analysis, it is 
difficult for CIDA to identify how it could make the best contribution. 


8.30 The means through which CIDA achieves harmonization in the 
field are important. Increasingly, donors are using coordination groups 
as a way to harmonize their efforts, providing fora to discuss the ways 
each can contribute to a country’s development. These groups also 
allow donors to engage governments with a collective voice to discuss 
programming. The structure, complexity, and membership of these 
groups vary from country to country. Typically, a number of sector- 
specific or technical sub-committees support an overall coordinating 
committee. 


8.31 These coordination groups are at different stages of development. 
For example, the coordination group in Honduras is struggling 

to maintain its effectiveness after major donors have withdrawn. 

In Vietnam, relationships have evolved to the point where some joint 
programs have been set up. In Ghana, donors, governments, and other 
stakeholders have signed a comprehensive formal joint cooperation 
strategy; whereas, in Bangladesh, they have committed to develop one. 
In Ethiopia, donors have started to map their respective projects and 
programs, working toward a goal of division of labour. A division of 
labour, whereby donors divide projects and responsibilities based on 
their areas of interest and relative strengths, is considered a long-term 
goal of harmonization and is increasingly being discussed in the donor 
coordination groups. 


8.32 Ultimately, the nature and effectiveness of such groups depends 
on the collective will and effort of all the donors. CIDA field staff put 
considerable effort into supporting these groups. Canadian-based staff 
and locally retained experts represent the Agency on a number of 
committees and working groups and, at times, take a leadership role. 
People we met with in development communities in the countries we 
visited described CIDA as a valued partner and active participant. 


Targets to guide the Agency’s use of program-based approaches have not been set 


8.33. The introduction of program-based approaches represented a 
significant departure from traditional ways of doing business. In the 
past, traditional aid projects were often carried out in isolation rather 
than designed to be part of a bigger picture. For example, building a 
school was the end product rather than part of a comprehensive 
education system. Moreover, traditional projects tend to rely on the 
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donor’s systems and procedures for delivery—rather than those of the 
recipient country—effectively impeding the development of national 
capacity. Finally, the cumulative effect of many donors implementing 
separate projects places a large administrative burden on recipient 
countries, who must cope with a myriad of rules and procedures 
imposed by the donors. 


8.34 Program-based approaches entail new forms of funding and can 
involve general budget support (donors transfer money to a 
government treasury for it to use according to its development and 
poverty reduction objectives) and sector budget support (money is — 
transferred but is targeted to selected ministries for specific programs). 
In some cases, donors first contribute funds into a pool or trust that is 
managed by an organization, such as a development bank, which 
subsequently transfers the money to a government or organization. 
We examined the extent to which CIDA supported the use of 
program-based approaches and the extent to which this was reflected 
in the Agency’s planning processes and documents. 


8.35 The country development programming frameworks for 
Bangladesh, Ethiopia, and Vietnam include general discussions about 
program-based approaches, highlighting how their use may address or be 
affected by certain risks in the country and indicating that the Agency 
will seek opportunities to use them. None of the frameworks contains 
specific targets concerning the desired level of use or mix of traditional 
and program-based approaches. In fact, in the early years after the 
release of its 2002 Policy Statement on Strengthening Aid Effectiveness, 
CIDA management provided little specific direction and no quantitative 
targets to country desks on how and when to use program-based 
approaches. Without these targets, the Agency cannot assess the extent 
to which it is meeting its own commitment and expectations. 


8.36 Nevertheless, program-based approaches are currently being used 
in each of the countries we examined, though the level of use varies 
significantly from country to country. In Ethiopia and Ghana, for 
example, multi-donor program-based approaches account for the large 
majority of CIDA’s funding; while in Honduras, CIDA funded only one 
such project. The extent of use of such approaches in Bangladesh and 
Vietnam was in between these two extremes. In part this is because of 
differences in the nature and capacity of recipient governments, the 
role other donors play, and the extent of support for their use within 
CIDA. Exhibit 8.3 provides an overview of the mix between traditional 
and program-based projects in each of the five countries we selected, 
as well as an example of a project from each country. 
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8.37 The program-based approaches CIDA is using vary significantly 
in design and funding arrangements. The Agency is using various forms 


‘of sector budget support in all of the countries we examined; it is 


providing general budget support only in Ghana and Vietnam. CIDA 
staff in the countries we examined are looking for opportunities to 
maintain or increase their use of the program-based approaches, largely 
through the donor coordination mechanisms that exist in each country. 


8.38 CIDA has now been using program-based approaches for almost 
a decade, and has gained considerable experience in designing and 
using them. We believe it is timely for the Agency to evaluate this 
experience to determine whether, in reality, such approaches are 
achieving their intended benefits and results and whether such 
approaches are right for CIDA. 


8.39 Recommendation. To support the use of program-based 
approaches, the Canadian International Development Agency should 
set clear performance targets regarding how much of its aid should be 
delivered through these approaches. 


The Agency’s response. Agreed. The Agency finalized an Aid 
Effectiveness Action Plan in July 2009, which identifies measurable 
actions with accompanying targets for aid effectiveness commitments, 
including program-based approaches. As committed in the Action 
Plan, the Agency will strive to meet its program-based approach target 
of 50 percent when appropriate conditions are met, as defined in the 
policy on program-based approaches. 


8.40 Recommendation. The Canadian International Development 
Agency should undertake a comprehensive evaluation of its use of 
program-based approaches with a view to determining whether they 
are achieving the Agency’s goals. | 


The Agency’s response. Agreed. As mandated by the Treasury Board, 
the Agency will provide a review of program-based approaches by 

31 March 2010 that assesses the effectiveness of these approaches. In 
preparation for this review, it is completing evaluations of country 
programs with a significant program-based approach component. 


A rigourous country planning process is missing 


8.41 In our view, the country development programming frameworks 
we examined served as useful planning documents at the time they 
were written. Analyses of recipient countries’ needs and extensive 
consultation with external stakeholders and within CIDA 
underpinned programming direction. The frameworks influenced the 
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selection of Agency-funded projects, including the use of program- 
based approaches. These public documents communicated CIDA’s 
plans and were the basis of alignment and harmonization discussions 
with governments and other donors. 


8.42 However, the framework process was never sustained as 
envisioned. For example, the frameworks for Ghana and Honduras 
(which were developed before the 2002 Policy Statement on 
Strengthening Aid Effectiveness) expired in 2004 and 2007, 
respectively. In both cases, CIDA staff assigned to these countries 
undertook renewed analyses and consultation, wrote new draft plans, 
and presented those plans internally for approval. They were never 
approved. Ethiopia’s framework was not updated to take into account 
significant changes that occurred in that country in 2005, changes that 
substantially altered CIDA’s programming there. The result was that 
stakeholders in Ghana, Honduras, and Ethiopia lacked formal plans 
that officially communicated the Agency’s direction. 


8.43 By the end of our audit, all of the frameworks for the 

five countries we examined had expired and none had been replaced 
with a comparable document. Some senior government officials and 
donor representatives that we met in the countries we visited remarked 
on the absence of a long-term CIDA plan for those countries. In effect, 
use of the country development programming frameworks was 
discontinued only a few years after they were put in place. 


8.44 In addition, although prepared by the Geographic Programs 
Branch, country development programming frameworks were intended 
to be corporate documents, reflecting the combined programming of 
CIDA’s Geographic Programs, Multilateral and Global Programs, and 
Canadian Partnership branches. In the countries we examined, CIDA 
is providing support through all three branches. In our view, 
harmonization with external parties demands internal coordination 
and cohesion. The frameworks we examined include a summary 
description of the activities of these other branches, but do not 
demonstrate a cohesive and coordinated approach to respective 
branch programming. Furthermore, the Agency has not developed the 
tools that could provide its managers—or its stakeholders—with an 
overview of all the Agency-funded projects and programs under way in 
any given country at any given time. Instead, CIDA country desks are 
left to piece together an integrated view of the Agency’s overall aid 


from various internal sources. 


8.45 Mid-2008 saw the emergence of a new planning process and a 
new type of country planning document, now referred to as Country 
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Strategies. During our audit, the Agency was still developing this new 
process and we observed confusion and frustration about it among 
staff. The purpose, format, required content, and even the title of these 
planning documents have been changed often. Staff were given little 
time to undertake meaningful consultation or robust sectoral analysis 
to support the strategic orientation of the new plans. By the end of this 
audit, the status of the new plans was unclear because none had 
received formal approval nor had been communicated broadly within 
CIDA. None had been made public. In the interim, project proposals 
submitted for corporate approval took into account factors such as 
previous strategic direction and Agency priorities, as well as 
conditions, needs, and emerging opportunities in the recipient country. 


8.46 The absence of a well-defined and transparent planning process 
and formally approved and public plans impedes communication with 
donors and recipient governments, leaving them unclear about the 
Agency’s direction and long-term commitment in individual countries 
or regions. This makes it difficult for the Agency to harmonize its 
efforts with other donors or to provide any level of predictability to its 
government partners. Furthermore, it does not provide staff with clear 
direction to propose projects strategically. 


8.47 Recommendation. To ensure transparent and predictable long- 
term planning, the Canadian International Development Agency 


should 


* finalize the new planning process for all countries included within 
its Geographic Programs Branch; 


* ensure that future planning is based on sound analysis, taking into 
account recipient country needs and capacities and CIDA’s 
relative strengths, and reflects an integrated whole-of-Agency 
perspective; 


* provide ongoing oversight of the process’s application and renew 
country plans, under defined and monitored conditions; and 


* communicate approved country plans to development partners, 
including recipient governments, as appropriate. 


The Agency’s response. Agreed. A new planning process for 
geographic programs has been finalized. Country Strategies for each of 
the 20 countries of focus, which provide the strategic direction for 
programs, have received ministerial direction. Summaries of these 
strategies will be posted on the Agency’s website by the end of 2009. 
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These documents contain an analysis of the recipient country needs as 
identified in their Poverty Reduction Strategy and National 
Development Plan. Canada’s value-added in addressing these needs is 
based on such factors as the Agency’s past experience in the country 
and its relative programming strengths compared to other donors and 
partners. The whole-of-Agency approach is also reflected, where 
appropriate. 


The direction identified in country strategies is implemented by the 
responsible program through five-year programming and planning 
frameworks, or Country Development Programming Frameworks 
(CDPFs), which will be completed by March 2010. They will be based 
on revised CDPF guidelines that will be put in place by the end of 
October 2009. 


Country strategies will be reviewed annually to assess any significant 
changes to the country’s development context and program 
performance. If required, adjustments will be made and reflected 

in changes to CDPFs, and the country strategies posted on the 
Agency’s website. 


Shifting and broad priorities have prevented a meaningful sector focus 


8.48 The Agency has recognized that its programming is widely 
dispersed. Over the years, it has tried to bring more focus to its 
programming by clearly defining sectors of priority. Canada’s 1995 
foreign policy statement set out six priority sectors for CIDA’s 
development program. The 2002 Policy Statement on Strengthening 
Aid Effectiveness noted that these priorities had served more as a broad 
menu of thematic options than as a focused agenda. This policy 
statement attempted to target CIDA’s investments more strategically. 
Also, in its reports on plans and priorities to Parliament, the Agency has 
set out to concentrate its resources on a smaller number of sectors to 
make a more meaningful contribution in the countries where it operates, 
given the size of its aid budget. We examined the extent to which the 
Agency has achieved this corporately and in its country-specific 
programs. Over time, CIDA has used a variety of terms to describe these 
priorities, including words like sectors, themes, and focus areas. For the 
purposes of this chapter, we are using the term “priority sectors.” 


8.49 As Exhibit 8.4 illustrates, CIDA’s sector-based policies and 
priority sectors shifted many times between 2001 and 2009. Often, the 
Agency announced new policies and priorities but did not rescind old 
ones. In some cases, these shifts represented a completely new 
direction for the Agency (such as the shift to agriculture as a funding 
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priority in 2003). In other cases, these shifts were more of an 
elaboration or restatement of previously announced priorities. Since it 
can take years to design and carry out projects properly and even 
longer for changes in direction to take hold, over time these frequent 
changes have hampered the ability of CIDA’s country desks to plan for 
the long term. Soon after the organization starts to adjust the nature of 
its programming, a change in direction or a new initiative begins, and 
staff must begin adjusting again. 


Exhibit 8.4 The Canadian International Development Agency’s (CIDA’s) priority sectors have shifted over time 


: Source document pla ee ee Prigritysoctire 


1995 Canada in the World— ¢ Basic human needs: primary health care, basic education, family 
Government Statement planning, nutrition, water and sanitation, and shelter 


e Women in development 

e Infrastructure services 

e Human rights, democracy, good governance 
e Private sector development 

e Environment 


2001 CIDA’s Social Development ¢ Health and nutrition 
Priorities: A Framework for Action ° Basic education 
¢ HIV/AIDS 


¢ Child protection 


2002 Canada Making a Difference in ¢ Four social development priorities: health and nutrition, HIV/AIDS, 


the World: CIDA’s Policy basic education and child protection, with cross-cutting theme of 
Statement on Strengthening Aid gender equality 
Effectiveness ° 


Agriculture and rural development 
e Private sector development 


2003 CIDA’s Policy on Promoting ¢ National capacity 
Sustainable Rural Development A 


Knowledge for development 
Through Agriculture 


¢ Food security, agricultural productivity, and income 
e Agricultural sustainability and natural resource management 
e Well-functioning markets 


2005 Canada’s International Policy * Governance 
Statement—A Role of Pride and * Health (focus on HIV/AIDS) 
Influence in the World: 


Development * Basic education 


° Private sector development 
¢ Environmental sustainability 
¢ Cross-cutting theme: gender equity 


2009 CIDA news release: Canada ¢ Food security 
Introduces a New Effective 
Approach to Canadian Aid to its 
International Assistance 


Sustainable economic growth 
Children and youth 
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8.50 CIDA’s priority sectors have always been broadly defined, which 
further complicated its attempts to focus. Identified priorities such as 
health, governance, or private sector development can encompass a 
wide range of and overlapping programming areas (sometimes called 
sub-sectors or activities). For example, the priority sector of private 
sector development encompasses more than 100 different potential 
programming areas. It is difficult to see how any greater focus can 
result from sectors as broadly defined as these. 


8.51 In the years immediately following the release of its 2002 Policy 
Statement on Strengthening Aid Effectiveness, the Agency did not 
develop a specific action plan to help it focus its aid in specific sectors. 
Nor did it set precise and measurable targets for country desks to 
achieve. In 2005, the Agency began an initiative to assess its 
intervention in a number of sectors, beginning with a stock-taking 
exercise. By the end of March 2006, the Agency had produced draft 
strategic direction papers intended to provide clear and consistent 
policy guidance for six priority sectors. The initiative as a whole was 
never completed. We note that yet another stock-taking exercise was 
taking place at the time of this audit. 


8.52 Our analysis of data the Agency provided us did not reveal any 
meanineful trends that would indicate that the Agency as a whole has 
been achieving a narrower focus since 2002. In addition, our analysis 
of individual portfolios in the countries we selected during this audit 
found that in four out of the five countries there was no significant 
narrowing of focus since 2002 in priority sectors. The portfolios of 
operational projects under way continue to cover a broad range of 
topics. The Agency has not provided us with its own analysis that 
would demonstrate otherwise. 


8.53 The lack of clear direction and action plans, coupled with broadly 
defined and shifting priorities, has led the Agency to a situation where, 
in our view, it is not realizing the benefit of its intended goal to focus its 
aid more narrowly. That goal is to create a more meaningful Canadian 
contribution to a country or region. Instead, the lack of direction has 
confused CIDA staff, recipient governments, and other donors, 
effectively undermining the Agency’s long-term predictability. Changes 
in priorities have little impact on projects that are already under way 
and have often meant that project wording, but not project design, is 
changed to fit the priorities of the day. 
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8.54 Recommendation. To achieve greater sectoral focus, 
the Canadian International Development Agency should 


¢ clarify, for each of its priority sectors, the specific programming areas 
that it will support in the future as well as those that it will not; 


* set measurable expectations and targets for country programs 
to achieve; and 


* monitor and report on progress against these expectations 
and targets. 


The Agency’s response. Agreed. To achieve greater sectoral focus, 
the Minister of International Cooperation announced new thematic 
priorities in May 2009. These thematic priorities are: increasing food 
security, stimulating sustainable economic growth, and securing the 
future of children and youth. 


The 2009 country strategies reflect greater programming focus on, and 
within, these thematic priorities. Following consultations in Canada 
and in focus countries, the Agency will complete Agency-wide 
strategies for each of these thematic priorities by early 2010. 


Measurable expectations and performance benchmarks will be set in 
comprehensive development programming frameworks as appropriate. 
The Agency will monitor and report on these expected results. Annual 
program performance reports will examine the extent to which key 
expected results have been achieved. These results will contribute to 
the formulation of corporate plans and reports, such as the Report on 
Plans and Priorities and the Departmental Performance Report. 


Lack of focus is impeding the identification and growth of the Agency’s 
relative strengths 


8.55 In our view, this situation has also affected the Agency’s ability to 
determine and build on its strengths relative to other donors and to 
identify and foster the sectoral expertise it needs for decision making 
and project design. Country desk staff typically operate as project 
managers and development generalists rather than sector specialists. 
Those who are specialists in a particular sector are not necessarily 
assigned to country programs where their expertise is directly relevant 
to those countries’ priorities. When necessary, CIDA relies on other 
in-house experts or international, Canadian, and local experts. While 
the 2002 Policy Statement on Strengthening Aid Effectiveness set out 
to help renew the Agency’s cadre of scientific and technical personnel, 
CIDA has made limited progress in deciding what type of expertise it 
requires to support its priority sectors, or how to provide it. This, 


Chapter 8 


Report of the Auditor General of Canada—Fall 2009 


Project selection and monitoring 


STRENGTHENING AID EFFECTIVENESS—CANADIAN INTERNATIONAL DEVELOPMENT AGENCY 


in turn, has implications for the Agency’s ability to define its areas of 
relative strengths and thereby support division of labour among donors. 


8.56 Recommendation. To support achieving sector focus and 
building its relative strengths, the Canadian International 
Development Agency should 


¢ define the skills and expertise it requires to support its priority 
sectors and specific programming areas; and 


¢ determine and implement the means it will use to provide those 
skills and expertise. 


The Agency’s response. Agreed. The Agency is in the process of 
defining the necessary skills and expertise required to support 
programming and delivery of initiatives within these thematic 
priorities. The Agency will ensure that a robust cadre of specialists is in 
place through recruitment, training and ongoing professional 
development. This work will be completed by March 2010. 


8.57 The benefits of Canada’s aid to developing countries depend on 


the projects that the Canadian International Development Agency 


(CIDA) funds. Agency-funded projects cover a wide range of 
programming areas. Although they can be one-time, short-term 
projects, they usually take place over several years. At any point in time, 
the Agency’s portfolio of projects in a country includes some that are 
just starting, some that are midway through implementation, and others 
that are nearing their completion. In order to maintain a targeted level 
of investment in a country, CIDA is always planning new projects. 


8.58 We looked at whether the Agency chose projects on the basis of 
a sound rationale; that is, whether its choices reflected the principles of 
alignment and harmonization, were consistent with the country 
development programming framework, and were based on CIDA’s 
strengths relative to other donors. We chose 5 projects from each of 
the 5 countries, for a total of 25 projects (about half of which were 
categorized by CIDA as using program-based approaches). 


Projects fit country needs 


8.59 The Agency’s rationale for choosing the projects we included in 
our selection varies widely. All of the projects contribute to needs the 
country has expressed; however, as noted earlier in this chapter, the 
needs are vast. In most cases, the projects align with the priorities 
expressed in CIDA’s country development programming frameworks at 
the time that it decided to fund them. Several projects are extensions 
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of previous ones that CIDA considered successful, often using trusted 
partners. The Agency chose other projects in response to explicit 
country requests. Still others resulted from invitations from other 
donors to join harmonized efforts. And while projects that are carried 
out through program-based approaches are—by their very nature— 
harmonized with other donors, we found that CIDA is also making 
efforts to harmonize with one or more donors and to promote some 
degree of local ownership for traditional projects as well. 


8.60 That said, the project documentation we reviewed does not 
generally show that CIDA chooses its projects on the basis of an 
assessment of its—or Canada’s—telative strengths in a particular field. 
It is not always clear what particular advantage (other than money) 
the Agency brings to the table. In our view, until sector focus is more 
clearly defined, choosing projects on the basis of CIDA’s relative 
strengths will be a challenge. Moreover, since the Agency’s country 
development programming frameworks have expired, and with shifting 
priorities and no up-to-date country plans, we are concerned about 
what CIDA’s basis will be for selecting future projects. 


Risks are assessed but the Agency needs to standardize its approach 


8.61 For both traditional projects and program-based approaches, 
CIDA’s decision-making and approval processes require that it identify 
project risks up front and put actions in place to monitor and, if 
necessary, mitigate those risks. We examined whether CIDA identified 
and assessed project risks before and monitored them after the project 
was under way. We emphasized projects that are carried out using 
program-based approaches because of the risks involved in relying on . 
national systems and in providing budget support to recipient 
governments. In such cases, special analyses are warranted, some of 
which are described in the guide on program-based approaches the 
Agency produced in 2005 (and updated in 2007) and in its 2007 policy 


on the assessment of fiduciary risks. 


8.62 For the projects we reviewed (most of which were initiated 

before 2007), the participating donors worked together to identify the 
types of assessment they needed. They carried out various types of 
analysis by examining public financial management and procurement 
systems, the level of corruption in a country, and the country’s capacity 
to carry out the program. The specific analyses undertaken and the 
analytic tools used varied from project to project. In some cases, the 
Agency hired its own contractors to assess the projects, but, for the most 
part, it relied on third-party assessments that other organizations (such 
as development banks) conducted. CIDA used the information from 
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these assessments to support its project decisions. Within the Agency, it 
also established an informal team of functional experts whose role was 
to assist staff responsible for particular countries in designing and 
implementing this approach to programming in Africa. At the time, 
most of the program-based approaches were taking place there. 


8.63 Notwithstanding the various assessments of risk that CIDA 

and others undertook, the Agency’s internal processes did not clearly 
stipulate the types of assessments that needed to be performed 
systematically before it agreed to become involved in a given program- 
based approach, nor did they clearly stipulate the criteria or conditions 
under which it would or would not participate in a program-based 
approach. At the outset of our audit, we were told that a new policy for 
program-based approaches—which would address these gaps—had 
been under development for several years. By the end of our audit, 

it had still not been approved. In our view, the Agency now needs to 
clarify its decision criteria and expectations for assessing risks and 
country Capacities. 


8.64 Recommendation. To support the use of program-based 
approaches, the Canadian International Development Agency should 


¢ define the specific conditions under which it will or will not 
participate in such approaches; 


¢ clarify the types of assessments of national systems and capacities, 
as well as of project-related risks, it requires to have undertaken, 
alone or in collaboration with other donors, prior to agreeing to 
use a program-based approach; and 


* institute processes to ensure the systematic and uniform 
application of such assessments. 


The Agency’s response. Agreed. The Agency finalized a Policy on 
Program-Based Approaches (PBAs) in July 2009. The policy 
articulates the circumstances under which it is appropriate to use 
PBAs. It is supported by the Agency’s Policy on Fiduciary Risk 
Assessment (Part 1), and Operational Guide on PBAs. These 
documents provide guidance to assess the capacity of national systems 
and identify assessment tools, such as the Public Expenditures and 
Financial Accountability diagnostic assessment. These tools enable the 
Agency to better identify and manage project-related risks. 

The Agency recently further clarified the approach and tools to assess 
and mitigate project-related risks in PBAs, by introducing updates in 
the Business Process Roadmap and Risk Management Strategy Guide 
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for Treasury Board submissions. The Agency will also institute a 
process to ensure the systematic and uniform application of the 
assessment tools by the end of 2009. 


Project risks are monitored 


8.65 Inthe planning documents for the projects we selected, CIDA 
had identified potential risks to the project as well as the means it 
would use to monitor them (and the implementation of the project 
overall) and measures it would take to mitigate them should they be 
realized. We found that the Agency carried out activities to monitor 
identified risks. Its monitoring can take various forms: 


* periodic progress reports, 

° field missions, 

* external evaluations, 

* participation in various specialized committees, and 


¢ informal communication with executing agencies. 


Executing agencies are generally required to complete periodic reviews 
at project milestones. CIDA may also conduct audits on projects to 
assess their compliance with the authorizing terms and conditions. 


8.66 Monitoring projects delivered through program-based 
approaches is particularly important, given the risks involved in relying 
on host country systems. In these cases, CIDA monitors jointly with 
other donors and the host government. In addition to reports from the 
executing agencies, donors rely on reports from the relevant 
government organization. We noted situations where CIDA 
contributed to designing ways to monitor the projects that were 
delivered through program-based approaches, and participated in 
harmonized monitoring efforts. 


8.67 At the end of each fiscal year, CIDA project officers must 
complete an annual project performance report for each project. 
These reports provide information about amounts disbursed, the 
expected and actual results, risk management, and lessons learned. 
In many cases however, the reports simply reproduce the list of risks 
and mitigating measures identified in the planning documents. In our 
view, reporting could be improved by more systematically including 
information about the current status of each potential risk and 
associated mitigating measures identified at the outset of the project. 
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Burdensome administrative processes hamper effective decision making 


8.68 Agency-funded projects are subject to a complex administrative 
and decision-making process. For traditional projects that are carried 
out through an executing agency, the process usually starts with a 
proposal or concept paper. This is followed by a project approval 
document and a logic framework analysis that describe the 
development context, the project and expected results, an estimated 
project budget, and anticipated risks and mitigating strategies. Both 
CIDA and the executing agency must then sign a contribution 
agreement or grant arrangement. CIDA and the recipient government 
also sign a memorandum of understanding. Once this document is 
signed, one of the partners or the group of those collaborating draws up 
a project implementation plan, which provides greater detail about 
how the project will be implemented. Then disbursements to the 
project can begin. 


8.69 The Agency’s decision-making processes have been widely 
criticized, and the Agency has acknowledged problems. The 2002 
Policy Statement on Strengthening Aid Effectiveness recognized that 
“over time, CIDA’s business processes have become unnecessarily 
complex and an inordinate amount of staff time has been taken up 
with managing the processes rather than making optimum use of 
development knowledge.” The Agency believed it would solve this 
problem by consolidating its business processes and applying new 
knowledge toward the delivery of development aid. However, this 
long-standing issue remains unresolved. | 


8.70 In 2007, the Agency undertook an internal study to improve 


the efficiency of its business processes. The study confirmed that the 


administrative burden remains serious and frustrating. It found that, 
on average, a project needed 28 different documents to take it from 
conception to the completion of final project implementation plan, 
and staff at times created their own tools to navigate through the 
process. They spent considerable effort on a process that took, on 
average, 43 months to get project approval. During this time, staff 
changes would occur and the same project officers would not see the 
majority of the projects through the process. Managers and sector 
specialists would also have moved, policy shifts would have occurred, 
and plans could have changed. The Agency advised us that, in 
response to the study, it has developed a new pilot project approval 
process, which it is currently evaluating. We strongly encourage the 
Agency to complete this evaluation on a timely basis and to implement 
appropriate actions to streamline its business processes. 
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Corporate management processes 
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and oversight 


8.71 According to responses to the 2008 Public Service Employee 
Survey, 88 percent of the staff in the Agency’s Geographic Programs 
Branch reported that the quality of their work often or almost always 
suffers.because of “too many approval stages.” The survey indicated 
that 62 percent of staff reported “unreasonable deadlines” had the 
same effect. These figures are more than double the average overall for 
the public service, which were 41 percent and 27 percent, respectively. 


8.72 The large majority of CIDA staff we interviewed and various 
members of the external development community also told us that 
administrative processes continue to burden the Agency. Many also 
commented on the lack of authority delegated to staff in the field. Levels 
of delegated authority have essentially remained unchanged for decades. 
They stated that CIDA staff in the field cannot commit to projects or 
actions before consulting with headquarters, which causes delays. 
Indeed, from a project approval standpoint, we note that all the planned 
projects included in the 2008 country strategies for the countries we 
selected required a higher level of authority than those currently 
delegated to the country directors. In fact, most of the projects required 
the Minister of International Cooperation or the Treasury Board to 
approve them. We have not assessed the extent to which these 
requirements contribute to lengthy decision making, but we appreciate 
that they need to be taken into account when planning large projects. 
Lengthy decision making can affect harmonization efforts in the field. 


8.73 The Canadian International Development Agency (CIDA) has 
been operating in an ever-shifting environment for many years, with 
changes to government policy objectives, emphasis on geographic 
orientation, priority sectors, and budget allocations. According to 
responses to the 2008 Public Service Employee Survey, staff in the 
Agency’s Geographic Programs Branch reported that “constantly 
changing priorities” (74 percent) and “lack of stability” (75 percent) 
often or almost always affect the quality of their work. These figures 
are substantially higher than the average for the public service overall, 
which reported 41 percent and 34 percent, respectively. In addition, 
CIDA also experiences significant changes in its senior managers from 
year to year. In its Geographic Programs Branch, since 2002, the 
average has been about 30 percent per year, and, in the Policy Branch 
(which develops policy for the Agency), it has been about 45 percent. 
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8.74 In light of this, management processes that can sustain 
implementation of commitments and directions over time are 
especially important. We examined whether management had 


* established specific action plans coupled with quantifiable targets 
and performance indicators; 


¢ assigned clear responsibility and accountability for 
implementation; 


* monitored progress of implementation and performance against 
expectations; and 


¢ made adjustments when performance was not meeting 
expectations. 


A comprehensive strategy for the Agency to achieve aid effectiveness is needed 


8.75 As described in previous sections of this chapter, CIDA has 
embraced the principles of alignment, donor harmonization, and the 
use of program-based approaches in its programs in the countries we 
examined. In our view, the release of the 2002 Policy Statement on 
Strengthening Aid Effectiveness has influenced this, bearing in mind 
that CIDA is working in an environment where the donor community 
at large is embracing these same ideas. 


8.76 After it put this statement in place, CIDA initiated some actions 
to align its geographic programming with aid effectiveness principles. 
However, these actions were selective and the Agency undertook them 
without a comprehensive strategy. In some cases, CIDA began 
initiatives but never tracked them to their completion. In other cases, 
the Agency never developed action plans at all. Much of the direction 
staff received from management was qualitative in nature. The Agency 
did not develop quantitative targets or measures even for areas where 
it made sense to do so, such as for the degree of sector focus it wanted 
to achieve or the level of projects it should fund through program- 
based approaches. There is little evidence that senior managers 
reviewed the implementation of the 2002 Policy Statement on a 
systematic basis. 


8.77 CIDA’s support for program-based approaches, in particular, has 
been neither uniform nor timely. While the Agency put some initiatives 
in place to guide the increased use of these approaches, at the time of 
this audit, other building blocks that we consider fundamental to the 
effective rollout of this new direction in delivering aid remained in draft 
form or had never been developed. Administrative and project approval 
requirements for this new way of delivering aid have not been updated. 
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8.78 In our view, many of the weaknesses described in this chapter 
can be traced to the absence of a master plan for achieving the 
Agency’s aid effectiveness commitments and the absence of 
management processes needed to sustain implementation over time. 
At the time of this audit, CIDA advised us of another renewed 
management effort toward policy implementation that was under way 
and that included an early draft of an aid effectiveness implementation 
plan. The Agency will have to ensure that it sustains any established 
management practices, and that the new effort can withstand some of 
the challenges we observed in this audit. 


8.79 Recommendation. The Canadian International Development 
Agency should establish and implement a comprehensive strategy for 
achieving its aid effectiveness commitments. Jo ensure that such a 
strategy can withstand changes in personnel and the passage of time, 
the Agency should 


¢ clearly document action plans, targets, and performance 
indicators; 


¢ assign clear accountability for their implementation; 


¢ evaluate and monitor progress against targets and performance 
indicators on an annual basis; and 


* communicate its strategy publicly and report on its 
implementation externally, as appropriate. 


The Agency’s response. Agreed. The Agency’s Aid Effectiveness 
Action Plan, approved in July 2009, consolidates guidance for meeting 
the Agency’s commitments to aid effectiveness. It is applicable to all of 
the Agency’s programs, as well as corporate and policy functions. It is 
accompanied by “Accountability and Monitoring Guidelines” that 
clarify accountabilities and tracking methodology for each action and 
target of the plan. Accordingly, monitoring will be performed annually, 
as a component of the Agency’s regular program monitoring process. 
Key elements of the plan and related progress will be communicated 
to Canadians through the Agency’s annual performance reports and 
its website. 
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Conclusion 


8.80 The Canadian International Development Agency (CIDA) has 
not put in place the required management processes to implement and 
monitor its aid effectiveness commitments concerning alignment, 
harmonization, the use of program-based approaches, and sector focus 
in the countries it focuses on. The Agency initiated some actions to 
align its geographic programming with aid effectiveness principles, but 
these actions were selective and the Agency undertook them without a 
comprehensive strategy. The Agency has not established benchmarks 
and performance targets that would allow it to assess the extent to 
which it is meeting its commitments. There is little evidence that 
senior managers reviewed the implementation of the 2002 Policy 
Statement on Strengthening Aid Effectiveness on a systematic basis. 


8.81 CIDA’s main documents for planning geographic 
programming—country programming development frameworks— 
reflected the principles of aid effectiveness when they were written. 
However, the frameworks for the countries we examined have expired 
and the process of preparing and maintaining them has been 
discontinued. The Agency is struggling to develop a new planning 
process. We were unable to conclude whether the new process will 
meet CIDA’s aid effectiveness commitments because the process is in 
transition and the status of interim plans is uncertain. 


8.82 CIDA’s project selection and delivery in the field has addressed 
recipient countries’ needs, and Agency personnel are actively 
promoting donor harmonization. The mix between traditional and 
newer programming approaches varies from county to country. CIDA 
field staff are looking for opportunities to maintain or increase their use 
of newer programming approaches, as appropriate. While the Agency 
can demonstrate a rationale for the projects it funds, its choices are not 
based on an understanding of its strengths relative to other donors. 
CIDA identified potential risks and mitigating measures in its project 
planning documents and carried out various monitoring activities for 


these risks. 


8.83 The long-term nature of international development requires 
stability and predictability of programming. In our view, in recent years, 
frequent changes in policy direction and substantial turnover of senior 
managers have posed significant challenges to CIDA in achieving its 
aid effectiveness agenda. 
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About the Audit 


All of the audit work in this chapter was conducted in accordance with the standards for assurance 
engagements set by The Canadian Institute of Chartered Accountants. While the Office adopts these 
standards as the minimum requirement for our audits, we also draw upon the standards and practices of 
other disciplines. 


Objectives 


The objective of the audit was to determine whether the Canadian International Development Agency 
(CIDA) applied the major principles and programming approaches of aid effectiveness it committed to 
when delivering assistance in selected countries of concentration. We expected that CIDA had 


* put in place adequate management processes to guide and monitor its commitments, 
* applied them in planning country programs;, 
- selected projects based on sound rationale and timeliness, and 


* monitored the implementation of projects. 


Scope and approach 


The major principles and programming approaches we selected for audit were drawn from the 
Agency’s 2002 Policy Statement on Strengthening Aid Effectiveness and the its reports on plans and 
priorities and included 


¢ alignment of aid with the countries’ needs and priorities, 
¢ harmonization with other donors and among branches in the Agency, 
¢ increased use of new programming approaches, and 


* greater sectoral focus. 
The audit was conducted at both CIDA headquarters in Gatineau, Canada and at missions abroad. 


For individual country planning and project audit inquiries, we concentrated our audit primarily on the 
Geographic Programs Branch country desks of Bangladesh, Ethiopia, Ghana, Honduras, and Vietnam. 
These 5 countries are on CIDA’s list of 20 countries of concentration. In comparison to that provided in 
fragile states or countries in crisis such as Afghanistan, Sudan, and Haiti, the aid that the Agency 
delivered in these countries was considered by us to be typical of the context in which it operates. We also 
considered geographic dispersion, the size of the aid program, and the use of a variety of programming 
instruments when we selected countries. We audited all 5 country programs at CIDA headquarters in 
Canada and, in addition, we visited Bangladesh, Ethiopia, and Honduras missions abroad. We examined 
how the Agency develops its long-term plans, taking into account the unique and specific context of the 
countries in which it operates. We examined various planning and project documentation. We conducted 
interviews with heads of aid and country program managers, various CIDA senior managers and staff, 
members of the program support units, recipient governments, other donors, and other stakeholders. 
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We selected 5 projects from each of the 5 countries to audit individually for a total of 25 projects. We selected 
the projects from information CIDA provided, with the criteria of having the first project disbursement in 
calendar year 2003 or afterward—subsequent to the 2002 Policy Statement on Strengthening Aid 
Effectiveness. We selected different sizes of projects, and those delivered by traditional and program-based 
approaches, across different sectors. Our project examination included review and analysis of documents; 
discussions with project officers; and, for those projects we visited, observations and discussions with project 
stakeholders, including executing agencies, government officials, other donors, and recipients. 


We excluded some aspects of the Agency’s aid effectiveness agenda from the scope of this audit, including, 
for example, the Agency’s commitments to enhancing its field presence (decentralization), to untying aid, 


and to concentrating on fewer countries. 


We did not audit project results because it takes a long time for benefits to emerge and because of 


measurement challenges and difficulties in attributing results to specific initiatives. These challenges are 
typical of the nature of development aid and not necessarily specific to CIDA. 


Criteria 


Listed below are the criteria that were used to conduct this audit and their sources. 


We expected that the Canadian International Development 
Agency (CIDA) would have established actions, targets, and 
indicators of performance for the 2002 policy statement 
commitments. 


¢ We expected that the Agency would have assigned clear 
responsibilities and accountabilities. 


¢ We expected that CIDA would have put in place practices to 
monitor progress and implementation. 


We expected that the Agency would have made appropriate 
adjustments to its implementation approach as needed. 


“We expected that the Agency's cou ntry plans would have taken 
into account key aid effectiveness principles and commitments: 
¢ alignment 

* harmonization 

* newer programming approaches 


* greater sector focus 


We expected that the Agency’s project selection would be 
consistent with country strategies and aid effectiveness 
principles. 


We expected that CIDA would have performed appropriate and 
timely monitoring and made adjustments, as required. 


¢ The Treasury Board of Canada Secretariat Management 
Accountability Framework 


° CIDA’s 2002 Policy Statement on Strengthening Aid 
Effectiveness 


* CIDA’s 2002 Policy Statement on Strengthening Aid 
Effectiveness 


¢ CIDA’s Guide for Preparing a Country Development 
Programming Framework 


* CIDA’s 2002 Policy Statem 
Effectiveness 


* Country Development Programming Frameworks 


Performance measurement frameworks, including project 
agreements 
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Management reviewed and accepted the suitability of the criteria used in the audit. 


Period covered by the audit 


The period audited for this chapter spans from September 2002, when the Agency’s Policy Statement on 
Strengthening Aid Effectiveness came into effect, to 15 May 2009. Review of individual projects was based 
on a selection of projects that were operational between January 2003 and December 2008. 


Audit work for this chapter was substantially completed 15 May 2009. 


Audit team 


Assistant Auditor General: Richard Flageole 
Principal: John Reed 

Lead Director: Dusan Duvnjak 

Director: Sharon Clark 


Robert Anderson 
Anthony Levita 

Catherine Martin 
Charlotte Stange 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 8. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


Planning and support of country programs 


8.39 ‘To support the use of program- 
based approaches, the Canadian 
International Development Agency 
should set clear performance targets 
regarding how much of its aid should be 
delivered through these approaches. 
(8.33-8.38) 


8.40 The Canadian International 
Development Agency should undertake 
a comprehensive evaluation of its use of 
program-based approaches with a view 
to determining whether they are 
achieving the Agency’s goals. 
(8.33-8.38) 


Agreed. The Agency finalized an Aid Effectiveness Action Plan 
in July 2009, which identifies measurable actions with 
accompanying targets for aid effectiveness commitments, 
including program-based approaches. As committed in the 
Action Plan, the Agency will strive to meet its program-based 


approach target of 50 percent when appropriate conditions are 
met, as defined in the policy on program-based approaches. 


Agreed. As mandated by the Treasury Board, the Agency will 
provide a review of program-based approaches by 

31 March 2010 that assesses the effectiveness of these 
approaches. In preparation for this review, it is completing 
evaluations of country programs with a significant program- 
based approach component. 


Report of the Auditor General of Canada—Fall 2009 


Chapter 8 35 


STRENGTHENING AlD EFFECTIVENESS—CANADIAN INTERNATIONAL DEVELOPMENT AGENCY 


36 


8.47 


To ensure transparent and 


predictable long-term planning, the 
Canadian International Development 
Agency should 


finalize the new planning process for 
all countries included within its 
Geographic Programs Branch; 


ensure that future planning is based 
on sound analysis, taking into 
account recipient country needs and 
capacities and CIDA’s relative 
strengths, and reflects an integrated 
whole-of- Agency perspective; 


provide ongoing oversight of the 
process’s application and renew 
country plans, under defined and 
monitored conditions; and - 


communicate approved country plans 
to development partners, including 
recipient governments, as 
appropriate. 


(8.41-8.46) 


Chapter 8 


Agreed. A new planning process for geographic programs has 
been finalized. Country Strategies for each of the 20 countries of 
focus, which provide the strategic direction for programs, have 
received ministerial direction. Summaries of these strategies will 
be posted on the Agency’s website by the end of 2009. 


These documents contain an analysis of the recipient country 
needs as identified in their Poverty Reduction Strategy and 
National Development Plan. Canada’s value-added in 
addressing these needs is based on such factors as the Agency’s 
past experience in the country and its relative programming 
strengths compared to other donors and partners. The whole-of- 
Agency approach is also reflected, where appropriate. 


The direction identified in country strategies is implemented by 
the responsible program through five-year programming and 
planning frameworks, or Country Development Programming 
Frameworks (CDPFs), which will be completed by March 2010. 
They will be based on revised CDPF guidelines that will be put 
in place by the end of October 2009. 


Country strategies will be reviewed annually to assess any 
significant changes to the country’s development context and 
program performance. If required, adjustments will be made and 
reflected in changes to CDPFs, and the country strategies posted 
on the Agency’s website. 
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8.54 ‘To achieve greater sectoral focus, 
the Canadian International 
Development Agency should 


* clarify, for each of its priority sectors, 
the specific programming areas that it 
will support in the future as well as 
those that it will not; 


set measurable expectations and 
targets for country programs to 
achieve; and 


monitor and report on progress 
against these expectations and 
targets. 


(8.48-8.53) 


8.56 To support achieving sector focus 
and building its relative strengths, the 
Canadian International Development 
Agency should 


¢ define the skills and expertise it 
requires to support its priority sectors 
and specific programming areas; and 


¢ determine and implement the means 
it will use to provide those skills and 
expertise. 


(8.55) 


Agreed. To achieve greater sectoral focus, the 
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mae 28, Geatees 
Minister of 
International Cooperation announced new thematic priorities in 
May 2009. These thematic priorities are: increasing food 
security, stimulating sustainable economic growth, and securing 


the future of children and youth. 


The 2009 country strategies reflect greater programming focus 
on, and within, these thematic priorities. Following 
consultations in Canada and in focus countries, the Agency will 
complete Agency-wide strategies for each of these thematic 
priorities by early 2010. 


Measurable expectations and performance benchmarks will be 
set in comprehensive development programming frameworks as 
appropriate. The Agency will monitor and report on these 
expected results. Annual program performance reports will 
examine the extent to which key expected results have been 
achieved. These results will contribute to the formulation of 
corporate plans and reports, such as the Report on Plans and 
Priorities and the Departmental Performance Report. 


Agreed. The Agency is in the process of defining the necessary 
skills and expertise required to support programming and 
delivery of initiatives within these thematic priorities. The 
Agency will ensure that a robust cadre of specialists is in place 
through recruitment, training and ongoing professional 
development. This work will be completed by March 2010. 
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Project selection and monitoring 


- 8.64 To support the use of program- 
based approaches, the Canadian 
International Development Agency 


should 


¢ define the specific conditions under 
which it will or will not participate in 
such approaches; 


clarify the types of assessments of 
national systems and capacities, as 
well as of project-related risks, it 
requires to have undertaken, alone or 
in collaboration with other donors, 
prior to agreeing to use a program- 
based approach; and 


institute processes to ensure the 
systematic and uniform application of 
such assessments. 


(8.59-8.63) 


8.79 The Canadian International 
Development Agency should establish 
and implement a comprehensive 
strategy for achieving its aid 
effectiveness commitments. To ensure 
that such a strategy can withstand 
changes in personnel and the passage of 
time, the Agency should 


¢ clearly document action plans, 
targets, and performance indicators; 


assign clear accountability for their 
implementation; 


evaluate and monitor progress against 
targets and performance indicators on 
an annual basis; and 


* communicate its strategy publicly and 
report on its implementation 
externally, as appropriate. 


(8.75-8.78) 


Agreed. The Agency finalized a Policy on Program-Based 
Approaches (PBAs) in July 2009. The policy articulates the 
circumstances under which it is appropriate to use PBAs. It is 
supported by the Agency’s Policy on Fiduciary Risk Assessment 
(Part 1), and Operational Guide on PBAs. These documents 
provide guidance to assess the capacity of national systems and 
identify assessment tools for national systems, such as the Public 
Expenditures and Financial Accountability diagnostic 
assessment. These tools enable the Agency to better identify and 
manage project-related risks. The Agency recently further 
clarified the approach and tools to assess and mitigate project- 
related risks in PBAs, by introducing updates in the Business 
Process Roadmap and Risk Management Strategy Guide for 
Treasury Board submissions. The Agency will also institute a 
process to ensure the systematic and uniform application of the 
assessment tools by the end of 2009. 


Agreed. The Agency’s Aid Effectiveness Action Plan, approved 
in July 2009, consolidates guidance for meeting the Agency’s 
commitments to aid effectiveness. It is applicable to all of the 
Agency’s programs, as well as corporate and policy functions. It 
is accompanied by “Accountability and Monitoring Guidelines” 
that clarify accountabilities and tracking methodology for each 
action and target of the plan. Accordingly, monitoring will be 
performed annually, as a component of the Agency’s regular 
program monitoring process. Key elements of the plan and 
related progress will be communicated to Canadians through the 
Agency’s annual performance reports and its website. 
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